{"id":844,"date":"2021-01-26T13:46:50","date_gmt":"2021-01-26T12:46:50","guid":{"rendered":"https:\/\/websites.fraunhofer.de\/video-dev\/?p=844"},"modified":"2021-01-26T13:46:51","modified_gmt":"2021-01-26T12:46:51","slug":"mse-a-story-of-append-windows-presentation-timestamps-and-video-buffers","status":"publish","type":"post","link":"https:\/\/websites.fraunhofer.de\/video-dev\/mse-a-story-of-append-windows-presentation-timestamps-and-video-buffers\/","title":{"rendered":"MSE: A story of append windows, presentation timestamps and video buffers"},"content":{"rendered":"\n

To some degree, we can compare a DASH MPD to a novel. A novel usually has multiple chapters (DASH periods) with exciting parts (high quality segments) and rather boring parts (low quality segments). Sometimes, we even skip some chapters (seek). Or, if you are like me and it is late (high presentation time), you lose concentration (your buffer is full) and fall asleep while reading (the end of the append window is reached). I admit, this was not the best analogy, but I could not think of a better transition to the main topic of this post: append windows, presentation timestamps, IDR frames and their impact on the media buffer.<\/strong><\/p>\n\n\n\n

In this blog post, we will examine how the position of Instant Decoder Refresh (IDR) frames and earliest presentation time (EPT) of a video segment interact can with Media Source Extensions <\/a>(MSE) attributes appendWindowStart<\/a>, appendWindowEnd<\/a> and timestampOffset<\/a>. Some of the fundamentals of this blog post are covered in more detail in our previous blog post “Common pitfalls in MPEG-DASH streaming”<\/a>. For now, we assume that the reader is familiar with the concept of adaptive streaming and the basic structure of a DASH Media Presentation Description (MPD).<\/p>\n\n\n\n

The append window <\/h2>\n\n\n\n

MSE-based DASH clients like dash.js and Shaka player use the appendWindowStart and appendWindowEnd attributes of the SourceBuffer object to define an append range. Coded frames with presentation timestamps within this append range are allowed to be appended to the SourceBuffer, while coded frames outside this range are filtered out. <\/p>\n\n\n\n

A DASH client typically aligns the MPD period boundaries with the append window. The start of the period will match appendWindowStart while the end of a period matches appendWindowEnd. That way, all media segments that are appended to the buffer are guaranteed to be included in a period. Frames that overlap the period boundaries are filtered out.<\/p>\n\n\n\n

In the basic example depicted below, the presentation timestamps of the first media segment are out of the append window. Consequently, the segment is filtered out. <\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Now, we can ask ourself what happens if we only partially<\/em> exclude a segment. For instance, see what happens if we only want to exclude the first part of the first segment:<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

The MSE provides us with a specific note for such use cases:<\/p>\n\n\n\n

Some implementations may choose to collect a few of these coded frames with presentation timestamp less than appendWindowStart and use them to generate a splice at the first coded frame that has a presentation timestamp greater than or equal to appendWindowStart even if that frame is not a random access point<\/strong>. Supporting this requires multiple decoders or faster than real-time decoding, so for now, this behavior will not be a normative requirement.<\/p><\/blockquote>\n\n\n\n

The significant part is marked in bold. A video segment usually has an IDR frame right at the start. In some cases where a single segment might contain multiple IDR frames, it is more common to include only a single IDR frame in one segment. So what actually happens if we try this out?<\/p>\n\n\n\n

The test setup<\/h2>\n\n\n\n

Test content<\/h3>\n\n\n\n

First, we generate some test content:<\/p>\n\n\n\n

Name<\/td>Seg length<\/td>EPT first segment<\/td>GoP size<\/td>FPS<\/td><\/tr>
2_sec_gop<\/td>6 sec<\/td>3.000\/90.000<\/td>60<\/td>30<\/td><\/tr>
6_sec_gop<\/td>6 sec<\/td>3.000\/90.000<\/td>180<\/td>30<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

<\/p>\n\n\n\n

Our test consists of two different configurations for the video segments. Both segment types have a length of six seconds and a framerate of 30 frames per second. We use closed GoPs for both segments. The main difference is the GoP size. The “2_sec_gop” segment has an IDR frame every two seconds. This results in three IDR frames per segment. On the other hand, the “6_sec_gop” segment only has a single IDR frame at the start. <\/p>\n\n\n\n

Interestingly enough, the encoder\/packager combination we used in this test sets the EPT value of the first segment to 3.000\/90.000. For some reason, this was not signaled in the corresponding MPD (no @presentationTimeOffset). In order to account for that, we can set the SourceBuffer@timestampOffset attribute to -3.000\/90.000. That way, we offset the EPT value in the segments and our buffer starts at 0. <\/p>\n\n\n\n

Test application<\/h3>\n\n\n\n

In order to check the buffer, we use the buffered attribute of the video element:<\/p>\n\n\n\n

 video.buffered.start(0) + '-' + self.video.buffered.end(0);<\/code><\/pre>\n\n\n\n
The MSE implementation is straight forward and not included in this post. Important changes to MSE attributes are explained in the respective test cases.<\/p>\n\n\n\n
Test platform<\/h3>\n\n\n\n
The following two tests were conducted on a Macbook Pro device with macOS Catalina 10.15.7 and the following  browser versions:<\/p>\n\n\n\n