Streaming is available in most browsers,
and in the WWDC app.
Explore HLS variants in AVFoundation
Discover how you can use AVFoundation APIs to highlight different variants of your content within your app. We'll show you how you can inspect HLS content using these APIs for different video characteristics, including attributes like SDR/HDR, FPS, and the like. And we'll explore the AVAssetVariant, which represents streaming and offline content.
- Have a question? Ask with tag wwdc21-10143
- HTTP Live Streaming - Overview
- Search the forums for tag wwdc21-10143
♪ Bass music playing ♪ Nishant Nelogal: Hello, and welcome to WWDC.
My name is Nishant, and I’m an engineer on the AVFoundation team.
Today, we will be looking at HLS Variants.
First, we will look at how you can inspect HLS variants, using AVFoundation APIs, and later on, we will see how we can use the HLS variants with downloads.
So let's start with variant inspection.
You all must be familiar with what a typical master playlist looks like.
Here is an example.
In this particular playlist, we have two variants.
One of the variants is an SDR variant with stereo audio, and the other one is a Dolby Vision variant with Dolby Atmos audio.
The asset representing this particular playlist may be presented something like this in your app.
Looking closely, you see that the asset has 4K, Dolby Vision, and Dolby Atmos badges.
Earlier, you had to get such information out of the band.
Now in iOS 15, you can directly inspect the HLS playlist to infer them.
For that, you start with an AVURLAsset pointing to the location of your master playlist.
Later on, you can obtain the HLS playlist via the variant's property.
This AVAssetVariant, as we see here, represents an HLS variant as is from the master playlist.
So it has multiple properties representing different media attributes.
Some of the attributes, such as the media bitrate, can be accessed directly.
Other attributes, such as those related to video and audio renditions, are grouped together in their own subclasses.
They are called as VideoAttributes and AudioAttributes, respectively.
As you see, each of them has relevant properties, which you can use to understand your asset.
Now you know how to inspect HLS variants in AVFoundation.
Let's see how we can use them with downloads.
Downloading HLS content for offline playback has been supported since 2016.
If you're not familiar with HLS download APIs, I would suggest you check out our earlier talk on the topic from WWDC 2020.
In iOS 15, we are taking our HLS download APIs and making them even more powerful.
Typically, you would like to influence the HLS variants selected for the download.
This may be due to business requirements, or you just want to provide more choice to your users.
Before, you could provide such input via downloadTask options.
We had an option for HDR, an option for lossless audio, and a few other attributes.
In iOS 15, we are expanding the variant selection with the use of NSPredicates.
You may be already familiar with using predicates from Core Data.
If not, don’t worry, you will learn about them today.
So to get started, let me introduce the variant qualifier interface.
This interface allows you to specify your variant preferences to AVFoundation.
And, as I said, they can be constructed using NSPredicates.
Let’s understand by looking at few examples.
Here, we have a NSPredicate which expresses peak bitrate to be less than five megabits.
You use this to construct your variant qualifier, which instructs AVFoundation to prefer variants under five megabits.
Simple enough, right? Let’s take a look at another example.
Here we create an NSPredicate for HDR video range.
And similarly as before, you can construct a variant qualifier for it.
You can also combine multiple predicates to create a compound predicate and use them to create your variant qualifier.
Any property on variant can be used to create a predicate against.
For properties such as audio channel count, which cannot be easily expressed using the predicate format string, we have custom constructors.
You can look them up in our header doc for variant qualifier.
Once you have your variant qualifier, you use it to create something called as content configuration.
Each content configuration represents a set of video, audio, and subtitle renditions.
OK, let's understand with an example.
Here is the variant qualifier which combines the two predicates we saw earlier.
Eh, this is compounding. Ha-ha.
It informs AVFoundation to prefer HDR variants under five megabits.
We also have a set of media selections representing English and French audio, and English subtitle renditions.
Both of these can be used to create a content configuration object.
You can create multiple of these content configurations and provide it to the downloadTask.
These multiple content configurations are tied together by the download configuration interface.
This is the root which holds everything together.
It is created with an AVURLAsset and needs an asset name, and optionally, an image.
The asset name and image are displayed in the Settings app.
This allows your users to manage all of their downloads in one place in the Settings app.
And of course, the downloadTask can be configured with multiple content configurations.
Looking closely, one of the content configurations will be designated as primary and the rest will be auxiliary.
The difference between the two is you typically want to download a primary set of video, audio, and subtitle renditions and then complement them with additional audio or subtitle renditions.
By specifying your additional renditions as auxiliary content configurations, you instruct AVFoundation to optimize and avoid downloading multiple video renditions.
It becomes more clear as we go over an example.
Here is the complete example.
First, we start by creating a download configuration with AVURLAsset and a title.
The primary content configuration is the same one we saw earlier.
It is configured to download HDR variants under five megabits, with English and French audio and English subtitle renditions.
In this particular example, we would like to complement it with an auxiliary content configuration to download English audio in lossless format.
Now we have both our content configurations, which we want to download.
Make sure to set optimizesAuxiliary ContentConfigurations to true.
By the way, it is true by default, and it allows AVFoundation to choose lossless variant, such that the lossless variant’s video rendition is the same as the primary content configuration.
Setting it to false may cause lossless variant to be evaluated independently, which may cause duplicate video renditions to be downloaded.
This may increase your download size, which we don’t want.
OK, once you have the download configuration, you can use it to create the downloadTask.
You resume the downloadTask to start the download.
Starting in iOS 15, you can also observe the progress of the downloadTask using NSProgress interface.
The NSProgress object is KVO-observable, and you can use it to update your user interface.
We understand that some of you may have business logic for selecting variants, which may be harder to express via predicates.
For such cases, you can also choose to choose the variants you want to download explicitly.
Here in this example, we have already picked our primary and auxiliary variant and the media selections to go with it.
We can instruct AVFoundation to download them as is, by creating a qualifier with the variant directly.
Be careful to select variants so that they are playable on the device where they are downloaded.
That's all we have today, folks.
To wrap up, we looked at how you can inspect HLS variants and use them to configure the downloadTask.
Along the way, we encountered different interfaces for downloadTask configuration.
The first one was variant qualifier.
This is used to express your variant preferences.
Then we saw content configuration interface.
The content configuration ties together your variant preferences and your media selection choices.
Finally, we saw the download configuration.
The download configuration is the root interface which ties everything together.
Last, but not the least, we also learned that you can monitor your downloadTask using NSProgress.
For more information, check our header docs.
They're very well commented.
Thank you for watching and bye-bye! ♪
0:40 - HLS Master Playlist
#EXTM3U #EXT-X-VERSION:7 #EXT-X-INDEPENDENT-SEGMENTS #EXT-X-MEDIA:TYPE=AUDIO,NAME="English",GROUP-ID="stereo",LANGUAGE="en",DEFAULT=YES, AUTOSELECT=YES,CHANNELS="2",URI="en_stereo.m3u8" #EXT-X-MEDIA:TYPE=AUDIO,NAME="French",GROUP-ID="stereo",LANGUAGE="fr",DEFAULT=NO, AUTOSELECT=YES,CHANNELS="2",URI="fr_stereo.m3u8" #EXT-X-MEDIA:TYPE=AUDIO,NAME="English",GROUP-ID="atmos",LANGUAGE="en",DEFAULT=YES, AUTOSELECT=YES,CHANNELS="16/JOC",URI="en_atmos.m3u8" #EXT-X-MEDIA:TYPE=AUDIO,NAME="French",GROUP-ID="atmos",LANGUAGE="fr",DEFAULT=NO, AUTOSELECT=YES,CHANNELS="16/JOC",URI="fr_atmos.m3u8" #EXT-X-STREAM-INF:BANDWIDTH=14516883,VIDEO-RANGE=SDR,CODECS="avc1.64001f,mp4a.40.5", AUDIO="stereo",FRAME-RATE=23.976,RESOLUTION=1920x1080 sdr_variant.m3u8 #EXT-X-STREAM-INF:BANDWIDTH=34516883,VIDEO-RANGE=PQ,CODECS="dvh1.05.06,ec-3", AUDIO="atmos",FRAME-RATE=23.976,RESOLUTION=3840x1920 dovi_variant.m3u8
3:38 - Peak bitrate cap predicate
let peakBitRateCap = NSPredicate(format: "peakBitRate < 5000000") let peakBitRateCapQualifier = AVAssetVariantQualifier(predicate: peakBitRateCap)
4:46 - Content configuration
let variantPref = AVAssetVariantQualifier(predicate: NSPredicate(format: "videoAttributes.videoRange == %@ && peakBitRate < 5000000", argumentArray: [AVVideoRange.pq])) let myMediaSelections : [AVMediaSelection] = [enAudioMS, frAudioMS, enLegibleMS] //English, French audio and English subtitle renditions let contentConfig = AVAssetDownloadContentConfiguration() contentConfig.variantQualifiers = [variantPref] contentConfig.mediaSelections = myMediaSelections
6:29 - Download configuration
let asset = AVURLAsset(url: URL(string: "http://example.com/master.m3u8")!) let dwConfig = AVAssetDownloadConfiguration(asset: asset, title: "my-title") /* Primary content config */ let varPref = NSPredicate(format: "videoAttributes.videoRange == %@ && peakBitRate < 5000000", argumentArray: [AVVideoRange.pq]) let varQf = AVAssetVariantQualifier(predicate: varPref) dwConfig.primaryContentConfiguration.variantQualifiers = [varQf] dwConfig.primaryContentConfiguration.mediaSelections = [enAudioMS, frAudioMS, enLegibleMS] //English, French audio and English subtitle renditions /* Aux content config */ let auxVarPref = NSPredicate(format: "%d IN audioAttributes.formatIDs", argumentArray: [kAudioFormatAppleLossless]) let auxVarQf = AVAssetVariantQualifier(predicate: auxVarPref) let auxContentConfig = AVAssetDownloadContentConfiguration() auxContentConfig.variantQualifiers = [auxVarQf] auxContentConfig.mediaSelections = [enAudioMS] //english audio dwConfig.auxiliaryContentConfigurations = [auxContentConfig] dwConfig.optimizesAuxiliaryContentConfigurations = true
7:42 - Download task
let myAssetDownloadDelegate = MyDownloadDelegate() let avurlsession = AVAssetDownloadURLSession(configuration: URLSessionConfiguration.background(withIdentifier: "my-background-session"), assetDownloadDelegate: myAssetDownloadDelegate, delegateQueue: OperationQueue.main) let asset = AVURLAsset(url: URL(string: "http://example.com/master.m3u8")!) let dwConfig = AVAssetDownloadConfiguration(asset: asset, title: “my-title”) ... let downloadTask = avurlsession.makeAssetDownloadTask(downloadConfiguration: dwConfig) downloadTask.resume() let progress = downloadTask.progress
8:10 - Direct variant selection
/* Example for direct variant selection */ let asset = AVURLAsset(url: URL(string: "http://example.com/master.m3u8")!) let dwConfig = AVAssetDownloadConfiguration(asset: asset, title: "my-title") /* Primary content config */ let myVariant : AVAssetVariant = ... let myMediaSelections : [AVMediaSelection] = ... let variantQf = AVAssetVariantQualifier(variant: myVariant) dwConfig.primaryContentConfiguration.variantQualifiers = [variantQf] dwConfig.primaryContentConfiguration.mediaSelections = myMediaSelections /* Aux content config */ let myAuxVariant : AVAssetVariant = ... let myAuxMediaSelections : [AVMediaSelection] = ... let auxVariantQf = AVAssetVariantQualifier(variant: myAuxVariant) let auxContentConfig = AVAssetDownloadContentConfiguration() auxContentConfig.variantQualifiers = [auxVariantQf] auxContentConfig.mediaSelections = myAuxMediaSelections dwConfig.auxiliaryContentConfigurations = [auxContentConfig]
Looking for something specific? Enter a topic above and jump straight to the good stuff.
An error occurred when submitting your query. Please check your Internet connection and try again.