Did some experimentation to find what the highest resolution and bit rate that could be put into the AVVideoCompressionPropertiesKey in an adjusted version of the ConvertingSideBySide3DVideoToMultiviewHEVCAndSpatialVideo project that Apple puts out.
Here are some experiment details:
- Video frames frames start off as individual PNGs that came out of the upscaler process. Each frame is about 100MB, at 16384×8192 (8192 per eye) side-by-side stereo
- Test video is output at 60 FPS
- Output video file is a MV-HEVC generated by the sample project from Apple, just modified to use the PNG frames instead of the video SBS input
- Variations to test are in sizes and bit rate:
- Resolution (per eye): 6k, 7k, 7.5k and 8k
- Bit rates (target-peak): 100-200, 200-300, 250-350, 300-400, 400-500)
- Videos will be 10 seconds long
And the goal of the experiment is to evaluate what the largest resolution and bit rate that can play in the Apple Vision without it going choppy or crashing.
For playing the videos on the Apple Vision Pro device, the https://github.com/acuteimmersive/openimmersive app was used.
Which is free and based off of SpatialPlayer by mikeswanson
The simulator is not enough to test this. Need the actual device.
Download the test files used here:
Initial tests were done with stream URL in OpenImmersive, however, to reign in more control over the test environment, went with the download 6GB zip, uncompress, and then load each file in OpenImmersive and see what happens.
Now for some results. Look at these file sizes for 10 seconds of video:
| Resolution | Bit Rate | Plays? | Notes |
| 8k | 100-200 | Yes | The Apple store logo gets blurry at this bit rate The trees don’t really have tree like qualities |
| 8k | 200-300 | Yes | The quality is better, maybe the windows on the right are getting a bit of compression artifacts? |
| 8k | 250-350 | Crashed | The Apple Vision crashed (similar to what was observed in our test apps as well) The whole OS seems to restart |
| 8k | 300-400 | Choppy | Plays for a few seconds, and then drops to 1 frames per second |
| 8k | 400-500 | Choppy | This plays for a few seconds, and then drops to like 1 frames per second of video |
So the 8k test is promising, seems 250/350 Mbit/s crashes it. Moving on to 7.5k
| Resolution | Bit Rate | Plays? | Notes |
| 7.5k | 100-200 | Yes | Same blurry trees and logo |
| 7.5k | 200-300 | Crashed? | It crashed at first, but was this because of an older video that had run? |
| 7.5k | 250-350 | Yes | Logo and trees don’t feel like the compression artifacts is making it blurry anymore |
| 7.5k | 300-400 | Yes | Also plays fine hmm |
| 7.5k | 400-500 | Crashed | OS Crashed after selecting the video |
| Resolution | Bit Rate | Plays? | Notes |
| 7k | 100-200 | Yes | Same as the other 100Mbit/s versions … trees are blurry, people are surrounded by compression artifacts |
| 7k | 200-300 | Yes | Smooth playback, feels good |
| 7k | 250-350 | Yes | Is the jump in bit rate visible in something? Maybe the smoothness of the surfaces of the buildings? |
| 7k | 300-400 | Crashed? | Re-opening it it and it played, but initial selection it crashed |
| 7k | 400-500 | Yes | Oddly this played |
So there is a bit of a weird thing here, where sometimes the OS crashes, and then after it crashes and re-open the same video and it plays fine.
Here’s a theory: When a bad video (e.g. too high resolution or too high bitrate) is played, the Apple Vision may not crash immediately, but if there is something newer queued for watching, remnants of that old bad video are somehow still in memory and makes the newer video (that would have otherwise not crashed) crashes.
Running theory. It’s like a bad video “pollutes” the OS and then the OS is just waiting for a little bit of a push to come toppling down and restarting.
Onwards with the testing!
| Resolution | Bit Rate | Plays? | Notes |
| 6k | 100-200 | Yes | Just like the others … but does the light of the Apple Vision in the store start to feel less smooth? |
| 6k | 200-300 | Yes | Feeling like the details are there without compromise |
| 6k | 250-350 | Crashed? | When clicking “Open from Files” the first time, it crashed without even loading the video. Hmm. It played afterwards. |
| 6k | 300-400 | Played | |
| 6k | 400-500 | Played |
So the conclusion here .. is a bit inconclusive? The player crashed, whether that was because of the video or not … unclear.
There is also the “choppy” behavior where it seems like it plays 1 frame per second, which may be a sign that the video player is going bad and that a OS restart is coming soon.
Testing does seem to show that true 8K (meaning 8000x per eye) makes it crash at 300+ bit rate pretty consistently.
In all these videos, at the 100M bit rate quality, the artifacts are very high. You can see what can be best described as “fuzz” that accumulates around people and foliage. Another way is trying to judge the “smoothness” of the lines and the surfaces to see if individual pixel edges can be observed, or if the buildings really do have indiscernible lines and walls.
Moving up to 200-300M bit rate, the artifacts start to simmer down.
Between the resolutions, it’s hard to grade how big of a perceptible gap is in between 6k 200Mbit/s and 8K 200 Mbit/s, or even 7.5K. 7.5K captures a balance of using all the pixels and not crashing the device.
Curious to learn what aspects of these videos should be looked at to make a better choice.
In closing, we think we will land somewhere between 200-300 Mbit/s and 7.5K for production use. Just below the limit where it does not seem to exhibit choppy playback, and balances preserving the details the upscaler is producing.
Consider that 100MB PNG file at 60FPS is 6GBPS per second? So if the video is 200-300 Mbit/s, that is a 20x reduction in file size? What happened to all those other bytes!
Thoughts for another day.