dash.js – Low Latency Streaming with CMAF

HTTP Live Streaming (HLS) and Dynamic Adaptive Streaming over HTTP (MPEG-DASH) are the two main formats used for adaptive streaming. While HLS is natively supported on most of its target platforms (iOS and MacOSX), external players are needed for MPEG-DASH. For browser-based environments, there are two great open-source options: shaka-player and dash.js. Both are written in JavaScript and use MediaSourceExtensions (MSE) and EncryptedMediaExtensions (EME) to enable playback directly in the browser without the need for external plugins. Both offer a wide set of features and have an active community. Shaka-player is maintained and developed by Google, while dash.js is the official reference player of the DASH Industry Forum. 

In this series of blog posts, I will focus on dash.js. I will explain how certain features are implemented and how they can be used within applications. This time, we are looking at a media file format called Common Media Application Format (CMAF), the mechanism it provides to do low latency live streaming, and how this works in combination with dash.js.

CMAF – Common Media Application Format

First of all, what is CMAF and why is it a huge step in the right direction? CMAF is essentially another media container based on ISOBMFF and fragmented mp4. The great thing about CMAF is that it can be referenced within HLS and DASH manifest files. In theory, we only need to encode and package our content once (within the CMAF container) and are then able to target all major platforms. We are no longer forced to create and store separate media files with a TS container for HLS and an ISOBMFF container for DASH. Thus, CMAF immediately cuts our storage and packaging costs in half while doubling CDN efficiency.
Additionally, CMAF even becomes mandatory when delivering 4k or HDR content with HLS because Apple does not support HEVC with TS containers.

However, some problems still remain. Legacy devices will most certainly not receive an update when adding CMAF support. Moreover, encrypting the content in a uniform way still remains an issue. Although CMAF supports Common Encryption (CENC), Apple devices using a Fairplay DRM require a different encryption scheme than most devices using Playready or Widevine DRM (CBCS vs. CENC encryption).

How CMAF enables low latency streaming

In addition to the aforementioned benefits, CMAF provides us with the necessary tools for low latency live streaming. There is a great talk on that topic by Will Law from Demuxed 2018. I highly recommend watching this video in order to get detailed insights on related technologies. For now, I will just pick up on the main facts in order to explain how low latency streaming works with CMAF.

CMAF Chunks to enable low latency streaming

CMAF introduces the concept of “chunks”. Think of a chunk as some kind of small segment inside the classic 2-6 second segments. On a high level, a classic segment consists of one “moof” box and one “mdat” box. With CMAF chunks, the segment now has multiple boxes that allow the client to access the media data before the segment is completely finished. The benefits of the chunked mode become more obvious when we look at a concrete example:

So let’s assume we have 8 second segments and we are currently 3 seconds into segment number four. For classic media segments, this leaves us with two options:

• Option 1: since segment four is not completed, we start with segment three. That way, we end up 11 seconds behind the live edge – 8 seconds coming from segment three, and 3 seconds coming from segment four.
• Option 2: we wait for segment four to finish and immediately start downloading and playing it. We end up with 8 seconds of latency and a waiting time of 5 seconds.

With CMAF chunks, on the other hand, we are able to play segment four before it is completely available. In the example above, we have CMAF chunks with a 1 second duration, which leads to eight chunks per segment. Let’s assume that only the first chunk contains an IDR frame and therefore we always need to start the playback from the beginning of a segment. Being three seconds into segment four leaves us with 3 seconds of latency. That’s much better than what we achieved with classic segments. We can also fast decode the first chunks and play even closer to the live edge.

CMAF low latency with dash.js

Let’s take a closer look how CMAF low latency works within dash.js (since version 2.6.8 dash.js has a low latency mode). Moreover, they offer two sample streams with low latency support generated by the DASH-IF live simulator. A concrete example of how to use dash.js in low latency mode is provided here.

Signaling low latency mode in the MPD

First of all, we need a way to signal to the client that our segments are chunked and available prior to completion. In the sample manifest files, we can identify two new attributes:

• @availabilityTimeComplete: specifies if all segments of all associated representations are complete at the adjusted availability start time. If the value is set to false, then it may be inferred by the client that the segment is available at its announced location prior to completion.
• @availabilityTimeOffset (ATO): provides the time in how much earlier segments are available compared to their computed availability start time (AST).

By setting @availabilityTimeComplete to “false”, we tell the client that the segments are available prior to being completed.

Using the @availabilityTimeOffset (ATO) attribute, we can specify how much earlier they are available. In our example, the segments have a duration of 8 seconds and the ATO is set to 7 seconds. This means that we have a chunk duration of 1 second and the first segment is available 7 seconds before its usual completion time.

Setting the target latency

The first thing we want to specify when streaming with dash.js in low latency mode is the target latency. The target latency defines how close we want to play to the live edge. In an ideal world, the target latency would be zero and we would play directly at the live edge. Unfortunately, reducing the target latency always come with the trade-off of reducing the size of the media buffer. A small media buffer, on the other hand, is very prone to bandwidth fluctuations. Worst case scenario, we run into an empty buffer and the playback stalls. Thus, always keep in mind that a small latency has a significant influence on the stability of your stream. We should always evaluate carefully before lowering the latency.

Setting the target latency with dash.js is very straight forward and only requires one line of code:

       player.updateSettings({
                'streaming': {
                    'liveDelay': 3
                }
            });

Configuring the catchup mechanism

In the example illustrated in Figure 1, we achieved a latency of 3 seconds for the CMAF fragment with a chunk duration of 1 second. However, there is an easy way to get even closer to the live edge: by simply increasing the playback rate. Increasing the playback rate is also useful in scenarios in which we encounter a latency deviation between our target and real latency.

In dash.js, this process is called catchup mechanism and there are two options in controlling it:

By setting the liveCatchUpMinDrift attribute, we can define the “minimum latency deviation allowed before activating the catch-up mechanism”. This value is specified in seconds. Again, the concrete call is a one-liner:

        player.updateSettings({
                'streaming': {
                    'liveCatchUpMinDrift': 0.05
                }
            });

In this example, we set the allowed drift to 0.05 seconds.

In addition, we need to define the desired catchup playback rate. Again, we only need one line of code for that:

    player.updateSettings({
            'streaming': {
                'liveCatchUpPlaybackRate': 0.5
            }
        });

In this example, the playback rate is set to 1.5 if the catchup mechanism is activated.

Personally, I feel that increasing or lowering the playback rate without informing the viewer can be irritating. For some viewers, it might look like the stream is broken, especially because the audio is affected as well. For that reason, you might want to implement a small notification pop-up window that informs the user about a change in playback rate.

Conclusion

This pretty much concludes our dive into CMAF, low latency streaming and the corresponding implementation in dash.js. Some final remarks: in order to use the low latency feature, the browser running dash.js needs to support the Fetch API and HTTP 1.1 chunked transfer encoding. The combination of both allows us to access the mediadata prior to the media segment being completely available.

Moreover, dash.js requires us to manually enable the low latency feature for a stream. This can be done directly in the options tab of the reference player or by setting the lowLatencyEnabled attribute.

If you have any question regarding our DASH activities or dash.js in particular, feel free to check out our website.