Stereo video HLS

I am trying to set up HLS with MV HEVC. I have an MV HEVC MP4 converted with AVAssetWriter that plays as a "spatial video" in Photos in the simulator. I've used ffmpeg to fragment the video for HLS (sample m3u8 file below).

The HLS of the mp4 plays on a VideoMaterial with an AVPlayer in the simulator, but it is hard to determine if the streamed video is stereo. Is there any guidance on confirming that the streamed mp4 video is properly being read as stereo?

Additionally, I see that REQ-VIDEO-LAYOUT is required for multivariant HLS. However if there is ONLY stereo video in the playlist is it needed? Are there any other configurations need to make the device read as stereo?

Sample m3u8 playlist

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:13
#EXT-X-MEDIA-SEQUENCE:0
#EXTINF:12.512500,
sample_video0.ts
#EXTINF:8.341667,
sample_video1.ts
#EXTINF:12.512500,
sample_video2.ts
#EXTINF:8.341667,
sample_video3.ts
#EXTINF:8.341667,
sample_video4.ts
#EXTINF:12.433222,
sample_video5.ts
#EXT-X-ENDLIST
Post not yet marked as solved Up vote post of altonelli Down vote post of altonelli
957 views
Posted by
altonelli
Reply
Add a Comment

Replies

Update on my end: I decided to using mediafilesegmenter Version 1.20.9 (618.21b-230614) to keep all software tooling Apple, however I ran into the following error:

Jan  1 2024 16:44:06.294: Fragmented MP4 is the only supported container format for the segmentation of HEVC content
Jan  1 2024 16:44:06.294: Unsupported media type 'hvc1' in track 0
Jan  1 2024 16:44:06.294: Unable to find any valid tracks to segment.
Segmenting failed (-12780).

Again, I'm trying to segment MV-HEVC (mp4) files which I converted using use the mvhevc1440x1440 AVOutputSettingsPreset with some overrides. My codec key is still hvc1 from the preset which makes sense for this error. What is confusing is that the hls recommendations (1.10) recommend hvc1. So I'm not sure why I'm getting an unsupported media type error.

Posted by
altonelli
Up vote reply of altonelli Down vote reply of altonelli
Add a Comment

I arrived at a solution. I came to realized that fMP4 (fragmented MP4) in the error was in reference to the output format rather than the source format. mediafilesegmenter defaults to ts segmentation rather than fMP4. I added -iso-fragmented option to the command so it looked similar to the demo script below.

mediafilesegmenter -iso-fragmented -t 4 -b https://www.example.com/spatial_video_stream/ -f ~/path/to/dir/with/file spatial_video.mov

Reference post that helped: https://developer.apple.com/forums/thread/71433

Posted by
altonelli
Up vote reply of altonelli Down vote reply of altonelli
Add a Comment

Did you figure this out? I'm getting the same error:

$ mediafilesegmenter -iso-fragmented -t 2 ./sample_spatial.mov       
Feb 17 2024 10:04:00.249: ISO fragmented mode, forcing segments to start with I-Frame
Feb 17 2024 10:04:00.249: Processing file /Users/jon/my_dir/sample_spatial.mov
Feb 17 2024 10:04:00.259: Format description for track index 3 mismatch with another track with same media type
Feb 17 2024 10:04:00.259: detected mismatch in track - don't use this file /Users/jon/my_dir/sample_spatial.mov
Feb 17 2024 10:04:00.259: Unable to find any valid tracks to segment.
Segmenting failed (-17185).

I'm using the latest mediafilesegmenter (Version 1.22 (659.13b-240124)

Posted by
jontg
Up vote reply of jontg Down vote reply of jontg
Add a Comment

Apple engineer mentioned in this video Deliver video content for spatial experiences where it says The HLS multivariant playlist format should be:

  • EXT-X-VERSION is updated to version 12
  • New REQ-VIDEO-LAYOUT attribute
    • For EXT-X-STREAM-INF tag for video content
    • Video Channel Specifier can have values CH-STEREO or CH-MONO or a mix

Firstly I tried using the simple command to parse a mv-hevc spatial video recored by a vision pro, but it ended up with an error "Unsupported media type 'hvc1' in track 0"

Then I tried @altonelli 's answer, i.e. added a param "-iso-fragmented", it worked.

mediafilesegmenter -iso-fragmented -t 4 -f output/sp_video-1-ip ./videos/sp_video-1-vp.MOV

Feb 22 2024 19:38:59.996: ISO fragmented mode, forcing segments to start with I-Frame
Feb 22 2024 19:38:59.997: Processing file /Users/sizhe/workspace/projects/avp-spatial-video/videos/sp_video-1-vp.MOV
Feb 22 2024 19:39:00.020: Finalized 2300 bytes in 0: /Users/sizhe/workspace/projects/avp-spatial-video/output/sp_video-1-ip/fileSequence0.mp4
Feb 22 2024 19:39:00.027: Finalized 14560188 bytes in 1: /Users/sizhe/workspace/projects/avp-spatial-video/output/sp_video-1-ip/fileSequence1.m4s
Feb 22 2024 19:39:00.027: Wrote full file size 14560188 to /Users/sizhe/workspace/projects/avp-spatial-video/output/sp_video-1-ip/fileSequence1.m4s
Feb 22 2024 19:39:00.027: segment bitrate 27.96 Mbits/sec is new max
Feb 22 2024 19:39:00.053: Finalized 14689506 bytes in 2: /Users/sizhe/workspace/projects/avp-spatial-video/output/sp_video-1-ip/fileSequence2.m4s
Feb 22 2024 19:39:00.053: Wrote full file size 14689506 to /Users/sizhe/workspace/projects/avp-spatial-video/output/sp_video-1-ip/fileSequence2.m4s
Feb 22 2024 19:39:00.078: Finalized 15339992 bytes in 3: /Users/sizhe/workspace/projects/avp-spatial-video/output/sp_video-1-ip/fileSequence3.m4s
Feb 22 2024 19:39:00.078: Wrote full file size 15339992 to /Users/sizhe/workspace/projects/avp-spatial-video/output/sp_video-1-ip/fileSequence3.m4s
Feb 22 2024 19:39:00.078: segment bitrate 28.76 Mbits/sec is new max
Feb 22 2024 19:39:00.103: Finalized 15668274 bytes in 4: /Users/sizhe/workspace/projects/avp-spatial-video/output/sp_video-1-ip/fileSequence4.m4s
Feb 22 2024 19:39:00.103: Wrote full file size 15668274 to /Users/sizhe/workspace/projects/avp-spatial-video/output/sp_video-1-ip/fileSequence4.m4s
Feb 22 2024 19:39:00.103: segment bitrate 29.38 Mbits/sec is new max
Feb 22 2024 19:39:00.128: Finalized 14482468 bytes in 5: /Users/sizhe/workspace/projects/avp-spatial-video/output/sp_video-1-ip/fileSequence5.m4s
Feb 22 2024 19:39:00.128: Wrote full file size 14482468 to /Users/sizhe/workspace/projects/avp-spatial-video/output/sp_video-1-ip/fileSequence5.m4s
Feb 22 2024 19:39:00.138: Finalized 4188748 bytes in 6: /Users/sizhe/workspace/projects/avp-spatial-video/output/sp_video-1-ip/fileSequence6.m4s
Feb 22 2024 19:39:00.138: Wrote full file size 4188748 to /Users/sizhe/workspace/projects/avp-spatial-video/output/sp_video-1-ip/fileSequence6.m4s
Feb 22 2024 19:39:00.139: average bit rate is 28.19 Mbits/sec - max file bit rate is 29.38 Mbits/sec

But when I looked into the m3u8 file, there is no REQ-VIDEO-LAYOUT attribute, neither the EXT-X-VERSION is 12 but 7. Any idea why?

#EXTM3U
#EXT-X-TARGETDURATION:4
#EXT-X-VERSION:7
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-INDEPENDENT-SEGMENTS
#EXT-X-MAP:URI="fileSequence0.mp4"
#EXTINF:4.16652,	
#EXT-X-BITRATE:27957
fileSequence1.m4s
#EXTINF:4.26653,	
#EXT-X-BITRATE:27544
fileSequence2.m4s
#EXTINF:4.26652,	
#EXT-X-BITRATE:28763
fileSequence3.m4s
#EXTINF:4.26652,	
#EXT-X-BITRATE:29379
fileSequence4.m4s
#EXTINF:4.26652,	
#EXT-X-BITRATE:27156
fileSequence5.m4s
#EXTINF:1.16663,	
#EXT-X-BITRATE:28724
fileSequence6.m4s
#EXT-X-ENDLIST
Posted by
brrruski
Up vote reply of brrruski Down vote reply of brrruski
Add a Comment