How on earth do you compress an MV-HEVC video file in-app without stripping the video of its 3D properties?

Hello @jakeadams1230,

The video settings you are using with your asset writer do not contain the keys/values that tell the asset writer that it should output as MV-HEVC.

For example, these are the video settings that the "Converting side-by-side 3D video to multiview HEVC" uses:

let multiviewCompressionProperties: [String: Any] = [kVTCompressionPropertyKey_MVHEVCVideoLayerIDs as String: MVHEVCVideoLayerIDs,
                                                          kVTCompressionPropertyKey_MVHEVCViewIDs as String: MVHEVCVideoLayerIDs,
                                              kVTCompressionPropertyKey_MVHEVCLeftAndRightViewIDs as String: MVHEVCVideoLayerIDs,
                                                   kVTCompressionPropertyKey_HasLeftStereoEyeView as String: true,
                                                  kVTCompressionPropertyKey_HasRightStereoEyeView as String: true
]
let multiviewSettings: [String: Any] = [AVVideoCodecKey: AVVideoCodecType.hevc,
                                                 AVVideoWidthKey: self.eyeFrameSize.width,
                                                AVVideoHeightKey: self.eyeFrameSize.height,
                      AVVideoCompressionPropertiesKey: multiviewCompressionProperties
]

Hey there @gchiste, thanks for getting back.

I implemented the proper video settings to tell the asset writer to output an MV-HEVC video, but upon testing this implementation, by passing an MV-HEVC video through my compressVideo function, my app crashes with the below error:

“Thread 3: "*** -[AVAssetWriterInput initWithMediaType:outputSettings:sourceFormatHint:] Compression property MVHEVCLeftAndRightViewIDs is not supported for video codec type hvc1"”

Here’s what my compressVideo function looks like when the crashing occurs:

let videoReaderSettings: [String: Any] = [kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32ARGB]
                let multiviewCompressionProperties: [String: Any] = [kVTCompressionPropertyKey_MVHEVCVideoLayerIDs as String: MVHEVCVideoLayerIDs,
                                                                     kVTCompressionPropertyKey_MVHEVCViewIDs as String: MVHEVCVideoLayerIDs,
                                                                     kVTCompressionPropertyKey_MVHEVCLeftAndRightViewIDs as String: MVHEVCVideoLayerIDs,
                                                                     kVTCompressionPropertyKey_HasLeftStereoEyeView as String: true,
                                                                     kVTCompressionPropertyKey_HasRightStereoEyeView as String: true
                ]
                let videoSettings: [String: Any] = [AVVideoCodecKey: AVVideoCodecType.hevc,
                                                   AVVideoHeightKey: videoTrack.naturalSize.height,
                                                    AVVideoWidthKey: videoTrack.naturalSize.width,
                                    AVVideoCompressionPropertiesKey: multiviewCompressionProperties
                ]
                
                let assetReaderVideoOutput = AVAssetReaderTrackOutput(track: videoTrack, outputSettings: videoSettings)
                let assetReaderAudioOutput = AVAssetReaderTrackOutput(track: audioTrack, outputSettings: nil)
                
                if assetReader.canAdd(assetReaderVideoOutput) {
                    assetReader.add(assetReaderVideoOutput)
                } else {
                    completion(.failure(NSError(domain: "VideoUploader", code: -4, userInfo: [NSLocalizedDescriptionKey: "Couldn't add video output reader"])))
                    return
                }
                
                if assetReader.canAdd(assetReaderAudioOutput) {
                    assetReader.add(assetReaderAudioOutput)
                } else {
                    completion(.failure(NSError(domain: "VideoUploader", code: -5, userInfo: [NSLocalizedDescriptionKey: "Couldn't add audio output reader"])))
                    return
                }
                
                let audioInput = AVAssetWriterInput(mediaType: .audio, outputSettings: nil)
                let videoInput = AVAssetWriterInput(mediaType: .video, outputSettings: videoSettings)
                videoInput.transform = videoTrack.preferredTransform

I have also tried multiple different implementations to configure the output settings for MV-HEVC (such as the one below), but I am still thrown the same error and my app still crashes.

let videoSettings: [String: Any] = [
                        AVVideoCodecKey: AVVideoCodecType.hevc,
                        AVVideoHeightKey: videoTrack.naturalSize.height,
                        AVVideoWidthKey: videoTrack.naturalSize.width,
                        AVVideoCompressionPropertiesKey: [
                        AVVideoAverageBitRateKey: bitrate,
                        kVTCompressionPropertyKey_MVHEVCVideoLayerIDs as String: [0, 1],
                        kVTCompressionPropertyKey_MVHEVCViewIDs as String: [0, 1],
                        kVTCompressionPropertyKey_MVHEVCLeftAndRightViewIDs as String: [0, 1],
                        kVTCompressionPropertyKey_HasLeftStereoEyeView as String: true,
                        kVTCompressionPropertyKey_HasRightStereoEyeView as String: true
                        ]
                    ]

I have also tried simply removing the line of code “kVTCompressionPropertyKey_MVHEVCLeftAndRightViewIDs as String: MVHEVCVideoLayerIDs”, but upon doing so my app continues to crash and I receive a different error relating to the configuration (the one below):

"Thread 7: "*** -[AVAssetWriterInput initWithMediaType:outputSettings:sourceFormatHint:] Compression property MVHEVCViewIDs is not supported for video codec type hvc1"”

I should also note that I'm working on an M3 iMac with sonoma 14.3.1 as my OS, so I don’t think hardware should be the issue. I’ve also tested multiple different MV-HEVC video files but they all crash as well.

Sorry for the long response, just figured I’d lay it all out to give out all the context I can. If there’s any changes I can make or anything you know of that will help fix this crashing error and allow me to compress MV-HEVC videos with the output also being MV-HEVC it would be greatly appreciated. Thanks.

Hey @jakeadams1230,

Thanks for the context! Those settings look okay to me, and indeed they work fine for me without error. Are you able to provide an example project that reproduces the error you are getting?

Accidentally posted my last reply as a comment... not sure it effects anything but I'm just going to re-post that same comment as a reply here just in case:

Thank you so much for continuing to help me out with this @gchiste, it really is so beyond appreciated. I went ahead and created a sample project that reproduces the error and crashing that I'm experiencing when trying to compress an MV-HEVC video with my current MV-HEVC configuration settings. I have attached the link to my sample project's zip file below.

My Sample Project (that reproduces the crashing error)

Hello @jakeadams1230,

Thank you for providing that sample project!

It turns out that the reason your code is crashing is because the MV-HEVC compression properties are not supported on visionOS, I'm sure that isn't the resolution you were hoping for, but at least it explains why you were encountering that error (my incorrect assumption was that you were running that code on macOS).

If you run this code, you will see that the MV-HEVC compression properties aren't supported on visionOS (they are not present in the dictionary):

        var session: VTCompressionSession!
        VTCompressionSessionCreate(allocator: nil, width: width, height: height, codecType: kCMVideoCodecType_HEVC, encoderSpecification: nil, imageBufferAttributes: nil, compressedDataAllocator: nil, outputCallback: nil, refcon: nil, compressionSessionOut: &session)
        
        var properties: CFDictionary!
        VTSessionCopySupportedPropertyDictionary(session, supportedPropertyDictionaryOut: &properties)
        
        print(properties)

So, my recommendation is that you file an enhancement request for these compression properties to be supported on visionOS using Feedback Assistant.

Thanks for getting back @gchiste

That really is unfortunate to hear. I'll go ahead and file a Feedback Assistant report.

Just as a last ditch effort prior to wrapping up our discussion... is there any way you know of that will allow me to compress MV-HEVC videos in-app on visionOS that doesn't involve the use of these compression properties? Any frameworks, techniques, libraries etc?

Hey @jakeadams1230,

Actually, good news, those properties are not supported in the visionOS Simulator, on an actual device, I am seeing support for the MVHEVC compression properties :)

Thank you @gchiste,

This is quite interesting. My intent with the MVHEVC compression properties is to be able to allow users to compress spatial videos (whilst preserving the stereoscopic/3D effect) in-app. Seeing as I do not own an Apple Vision Pro device myself, the simulator is all I have to work with.

And given your claim that you are seeing support for the MVHEVC compression properties on a real device, I would like to push this update for my app (that includes this compression functionality for spatial videos), but I'm a bit unsure how to go about it, seeing as I'm unable to tell from my end using the simulator whether my compressVideo function (the one from my sample project) actually works as intended without crashing and whilst preserving the stereoscopic effect of the original uploaded MV-HEVC video.

Any advice on how to go about this? Thanks again!

Hello @jakeadams1230,

The "Running your app in Simulator or on a device" article states, "To verify your app runs exactly as intended, run it on one or more real devices."

One way to do that is in an Apple Vision Pro Developer Lab!