What's New in TVMLKit

Hi everyone. My name is Jeremy and I'm an engineer on the tvOS team. Today I'm really excited to speak with you about what's new in TVMLKit that we've added in tvOS 12. For the uninitiated, TVMLKit is Apple's high level framework for quickly building content based applications on tvOS. Out of the box, it is compliant with our human interface guidelines. So your applications look beautiful and feel correct. It uses JavaScript to drive application logic and a special XML based markup language which you define that renders into user interfaces on the screen.

In fact, some of the apps you're familiar with and use today are built using TVMLKit. Along with that, there are thousands more in the app store that you write.

So let's get started with the enhancements we've made to TVMLKit, beginning with three things we want to talk about today. First, we have some enhancements to Web Inspector that allow you to further debug your applications and introspect them at a deeper level.

Since last year, we've added a bunch of features and enhancements to the framework and I'm going to discuss three of them with you today. We're going to talk about new Web Inspector enhancements for better debugging, new data binding constructs that are added to make it more powerful, and finally, a new way to customize the playback experience on TVMLKit. Let's get started with Web Inspector. tvOS 11 introduced more support for Web Inspector that allow better introspection into your TVMLKit apps. Since then, we've enhanced this support even further. In tvOS 11.3, in addition to showing install event listeners on an element, you can now temporarily disable them. This vastly helps with debugging as you can now toggle the event handlers off and on at whim.

In the network tab, joining at in document resources are image resources. This allows you to see the images that are being loaded as well as information about how long it took and where time was spent. If you are interested in seeing the actual image that comes off the wire, this is now also an option you have at your disposal. However, please note that this only works after you've connected the Web Inspector. It does not show what has already been loaded.

Finally, my most favorite feature of all, the inspect button. Clicking this will show you the approximately element that represents the view that's currently in focus.

If your elementary is collapsed, Web Inspector will expand it to the exact element and highlight it.

To use Web Inspector, either download the latest version of macOS and install it, or use Safari technology preview. More information about how to use Web Inspector can be seen in our talk given last year using Web Inspector with tvOS apps.

With that, let's talk about data binding.

And before we jump into the new features of data binding itself, let me give you an overview of what it is. Data binding helps you easily transform your data to user interface elements through expressions in your template markup. This is really good because it allows you to decouple your data and your lot -- and your layout logic as well as your application logic itself.

Because of this, data bound templates can reduce the amount of JavaScript code you actually have to write to convert your data to TVML documents, since the framework does this on your behalf. In fact, by authoring your documents on your behalf, it is able to do it in the most performing way so you don't have to worry about the right APIs to use. Let's look at a practical example to explain this. Say you want to generate a banner with a title and a description. And this is how you would typically do it without data binding itself.

First, you will fetch the relevant data that's supposed to be shown to the user in this case, a title and a description.

And upon fetching, you pass it onto a part of the JavaScript code you've already written that processes this data and generates the actual final document itself.

Now, with data bindings, you can take out JavaScript processing step and provide binding specifications in your template itself, and TVMLKit will populate the final document on your behalf as directed. Effectively, your application just needs to worry about fetching and massaging data, and not worry about dom editing at all. In a nutshell, that's how data binding works and helps reduce the amount of code you write.

Last year, we introduced the concept of data binding with some core concepts that include binding of attributes of an element, binding the text content of an element, and of course, binding items of a section in either a shelf, grid, or list. Let's quickly recap on these concepts with another example.

The corresponding data bound template representation of this image element would contain a binding expression with the 'at' symbol, followed by the attribute name and the property you want to bind it to. Next, let's seek the example of generating text content of an element.

Like in this case, a title element that has corresponding data of what it should actually be filled with.

The data bound template representation for this title element will contain a text content binding and the property it maps to.

Finally let's talk a little bit about items binding. It is a slightly different binding and involves a group of data pieces you want to show. It also only works with sections on a shelf, list, or grid.

In this example, we have some data with the tree list items as an array. And the final representation should be a section of tree list item lockups.

The corresponding data bound template for this section will contain two things; an item's binding and the property it maps to, and also a prototype to convert each data object contained in the array. In this case, it would be a data bound template representation of a list item lockup.

So that's the tree binding constructs we've introduced in tvOS 11. For more information, please look at our talk, Advances in TVMLKit from last year's WWDC.

This year, we've extended that vocabulary. To begin, we've added a more generic way of binding child elements of an element using the children binding.

And to help better organize your DOM, we've added a couple of special elements, namely fragment and rules. We'll go into this in detail, starting with children binding.

Children binding is a more generic form of items binding. Items binding is optimized for use cases where you have a section of a shelf, grid, or list in order to work efficiently with large data sets. They can be used outside these elements. For everything else, use children's binding.

And it's because of a very simple reason. It works by generating children of a target element itself.

And it behaves the same way as items binding does. You require the use of prototypes to define the elements that data should be converted into and this will be used as a template when generating the final DOM itself.

Let's take an example to explain how this works. I am going back to the same example of tree items in an array. We have this very same piece of data but in this particular situation, we have three different menu items. And this would be used in a menu bar. This is the final representation we expect to see. Basically a menu bar tree, menu bar items.

And this is a very simple way to specify the template for it. As you can see, it's similar to how you would do items binding. It has prototypes that's used to map data to elements and it has a binding expression. The only difference is that the element is expressed on section. Children binding works with any element.

Now this works really well as long as you want to generate all the child elements itself. There might be cases where you want to only generate some of those children. Take, for example, an auto streaming app with a now playing menu item. The now playing menu item is a special menu item that should always be in the menu bar but would only appear if background audio is currently playing.

However in this case, we still want the menu bar items to be data bound. In order for this work, we need to compartmentalize the data driven and non-data driven parts of the menu bar. And that's where fragments comes into play. So what are fragments? Fragments are invisible elements that the renderer doesn't see and it helps you organize your DOM. But what's special about fragments are that its children are visible.

And because it is an element and children binding works with any element, the fragments itself works with children binding.

So let's go back to our data in the final form we want and we have something like this. You have the menu bar items. Nice to encapsulate in a fragment and this is great because it allows us to do children binding like so.

Now all we have done is moved the data around into fragment while keeping the menu bar intact with the now playing menu item. And because the renderer only sees the children of the fragment, this still renders as a properly formed menu bar.

On the very subject of data itself, that you can use to map to user interface elements, this is another likely scenario where you have data that some do not change, and some change all the time. For example in this particular piece of data itself, it is for a video that has a poster image, a title, and a playback progress.

In some situations, we want to show different user interfaces based on that information itself. In the case where playback progress hasn't started, it's naturally zero and we are interested in showing the poster image and the title for the video itself.

But when we start watching the video, progress will naturally be greater than zero. For that, we want to show the same thing that never change, the poster image and its title. However to make things obvious this video's being played back, we want to show a progress bar that's filled through the playback progress' percentage itself.

In the very same vein, we have now two use cases where data is different and we also want to show two different looks of what it is. In the first case, we -- all we have is an image and a title in the lock up. And in the second case, we have the addition of an overlay and a progress bar.

Typically you would have application logic that outputs different x amount based on that data itself but with rules, you can have a single statically defined template that would give you either representations of the final document.

So what are the rules? Well, rules used data states to redefine your final document which, in turns, refines your UI. It is an invisible element. So the renderer doesn't see it but it affects how the document is authored.

Any operations within those rules happen on sibling elements that the rules are a part of. And the best way to show this to you is in another example of how this can be setup.

So let's look at the rules that we would use to structure the very prototype that we want -- that we've shown as an example. We will start by defining the prototype as the lowest common denominator of what our user interface should look like. And in this case, we have an image and a title.

However, you would notice that we also have a place holder for the progress bar. Placeholders are also special elements that are invisible to the renderer and in this case, would be used by the rules as a target for replacement when data states match.

Now let's fill in our rules. A group of rules that act on sibling elements are encapsulated in the rules tag.

Individual rules that match a data state are encapsulated as specialized elements, and specialized elements become active when it matches certain data states. And the way that that's matched is based on a query in the state attribute itself.

When data state matches, the children of the special element -- specialized element are the list of operations that act on the sibling elements of rules. In this case, we want the placeholder to be replaced with an overlay element and its children.

TVMLKit matches the elements to be replaced by looking at the tag attribute on any element. It effectively does replacement by first matching on that tag and then comparing the element name. If the element name is different, it would replace that element hole and in this case, placeholder becomes an overlay. However, if the element name matches, whatever that's new would be appended to what's already existing.

And so we have a single template with rules that will generate two different output based on the state of the data that's provided.

Effectively you can move your application logic to deal with how things are displayed into a statically specific template that exists within the context of elements that's transformed into user interfaces. Now let's switch gears and talk about playback in TVMLKit. TVMLKit has long provided extension points where you need more customization of your user interfaces, be it individual views or even whole templates. In tvOS 12, we are extending this to our playback pipeline, giving you control over playback as well as its associated user experience.

This experience works with all the different playback styles we have, whether it is embedded or in a full screen.

You do this by providing a TVPlayer object and its associated user interface as a UIViewController.

These have close analogues to our JavaScript APIs. So that would be less confusion in talking to your JavaScript developers. And finally there's a limited JavaScript Bridge that's exposed. It allows communications between your native code and JavaScript itself.

Let's talk about TVPlayer and that's the base where you can get your customized playback experience working. TVPlayer is a public AVPlayer adaptor to the Playback Pipeline. What this means is that TVPlayer effectively translates regular AVPlayer callbacks into what JavaScript expects.

TVPlayer is also the object that you can use to dispatch custom events to JavaScript and by default, it already handles everything that AVPlayer has as playback events. So anything extra is on you to dispatch.

Changes that JavaScript makes to the player are KVO observable. So you know when things are changed by your JavaScript developers. Finally the TVPlayer object plays media in a sequential fashion from the very first item all the way to the very last in its playlist.

Whenever a player is needed, TVApplicationControllerDelegate will ask for a TVPlayer and you will have to return an instance in order to participate in the Playback Pipeline.

The next step of the Playback Pipeline is to actually show Playback on the screen itself in the form of a user interface. This can happen anytime whether it's full screen playback or embedded playback. It is entirely up to you to create your own user interface. When TVMLKit needs a user interfact, the TVInterface will ask its delegate for a view controller. It will also pass it a reference to TVPlayer that is responsible for playing media in that view.

With everything, there are several caveats to using TVPlayer and its associated user interface.

The very first thing is you should handle any 'should' events yourself. These are usually tied to interstitials and since that's largely user interface, this should be handled by the native code that you write.

If you use FairPlay encryption for your video playback, you need to use AVContentKeySession for secure key loading. For more information about AVContentKeySession, look at our talks from last year, Advances in HTTP Live Streaming, and a talk from this year about AVContentKeySession Best Practices.

Finally, if your JavaScript developers use overlay and interactive overlays, this will not work out of the box. They are user interfaces and because you are building your own, you will have to handle it yourself.

In summary, the changes we've made to TVMLKit and tvOS 12 are the following two: One is the data binding is now even more powerful, enabling you to build data driven templates in any form. We highly encourage you to check this out.

And finally, if you've been long waiting to customize the playback experience itself, you can do it today by implementing your own native playback experience.

For more information about this talk, please look at the following url.

And thank you for attending WWDC 2018. Thank you.