{"id":1600,"date":"2023-10-31T15:56:57","date_gmt":"2023-10-31T14:56:57","guid":{"rendered":"https:\/\/websites.fraunhofer.de\/video-dev\/?p=1600"},"modified":"2023-11-09T09:10:05","modified_gmt":"2023-11-09T08:10:05","slug":"being-trapped-in-a-gap-with-big-buck-bunny","status":"publish","type":"post","link":"https:\/\/websites.fraunhofer.de\/video-dev\/being-trapped-in-a-gap-with-big-buck-bunny\/","title":{"rendered":"Being trapped in a gap with Big Buck Bunny"},"content":{"rendered":"\n

Introduction<\/h2>\n\n\n\n

When I thought about how to introduce this blog post, the first thing that popped to my mind was to start with a warning. Something like \u201cthis blog post will be technical and focus on details that are only be relevant for player developers\u201d. While the latter is probably still true, I am not sure anymore if I can call it \u201ctechnical\u201d after watching the Demuxed presentation<\/a> by Matt Szatmary (Mux).<\/p>\n\n\n\n

In any case, just a few days ago I stumbled upon a GitHub issue<\/a> that was raised in 2016 by David Evans<\/a>, a long-time contributor to the dash.js project.<\/a> The issue deals with a limitation of the Media Source Extensions (MSE)<\/a> that probably every media player developer has at least heard of: \u201cPlaying through unbuffered ranges and allowing the application to provide a buffered gap tolerance\u201d.<\/em> To the few readers that made it until this part of the blog post and have not heard about this limitation: First, thank you for not closing this tab and second, you are in for a real treat. Let\u2019s look at a concrete example of what we mean by \u201cunbuffered ranges<\/em>\u201d and \u201cgaps<\/em>\u201d.<\/p>\n\n\n\n

Gaps in the media buffer<\/h2>\n\n\n\n

When using the MSE we need to create SourceBuffers<\/a> to which we append our media segments. The SourceBuffer object supports two append modes<\/a> namely \u201csegments<\/em>\u201d and \u201csequence<\/em>\u201d. Typically, the \u201csegments<\/em>\u201d mode is used, and the media segments are placed in the media buffer according to their presentation timestamps. While this allows us to append the media segments in an arbitrary order, it can also cause non-alignments as illustrated below.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

In this example, segment 1 and segment 2 are perfectly aligned, but there is a gap between segment 2 and segment 3. There are various reasons for these kinds of gaps, such as media segments being shifted out of a DASH period (negative @eptDelta<\/em>), wrong presentation timestamps in the media segments or the total sample duration of a media segment not matching its duration. In “MSE: A story of append windows, presentation timestamps and video buffers<\/a><\/em>” we examined such situations in more detail.<\/p>\n\n\n\n

Regardless of the reason for the gap, native MSE implementation in all major browsers (Chrome, Firefox, Edge, Safari) stall at such gaps until they are filled with data. The problem is that in most cases there is no data to fill the gaps as they are usually not created intentionally. Due to that reason, media players such as dash.js and Shaka player implement a logic to detect and handle such gaps to continue the media playback.<\/p>\n\n\n\n

Native gap handling of the MSE<\/h2>\n\n\n\n

Coming back to our original Github issue, the suggestion that Dave made was to leave the handling of the gaps up to the browser and expose an API to control the behavior. Unfortunately, this is still an issue today and there is no native solution for the problem. Matt Wolenetz<\/a> (previously Google, W3C Invited Expert) pointed out that the issue\u00a0was discussed again at the FOMS 2023. The second part of Matt`s comment is where it gets interesting:<\/p>\n\n\n\n

\n

Meanwhile, there is an ugly hack that might help coalesce unintended gaps in Chromium SourceBuffers: the coalescence heuristic currently used there is based on the largest frame duration buffered so far in that particular track in that particular SourceBuffer. Auto-coalescence of gaps can thus be achieved by manipulation of the inputs to this heuristic:<\/p>\n\n\n\n

    \n
  1. For each track, first append a simple keyframe with huge duration. Ensure it is buffered. Then remove it from the SourceBuffer (or overlap it in your first appends to that track).<\/li>\n\n\n\n
  2. This should trick the heuristic.<\/li>\n<\/ol>\n<\/blockquote>\n\n\n\n

    A native solution for Chrome and other browsers?<\/h2>\n\n\n\n

    Now we are curious to try this workaround. Not only to verify that it is really working in Chrome, but there is also the slight hope that it might work in other browsers such as Firefox and Safari as well.<\/p>\n\n\n\n

    A simple MSE player<\/h3>\n\n\n\n

    The first thing we need is a simple standalone MSE player. We would rather not implement this \u201chack\u201d in dash.js or Shaka. It simply creates lots of overhead and requires us to change several classes. Our simple MSE player looks like this and supports the basic append and remove operations (Note I am not pasting the whole source code here but just the important parts).<\/p>\n\n\n\n

    App.prototype.createMediaSource = function () {\n    var self = this;\n\n    self.mediaSource = new MediaSource();\n    self.video.src = URL.createObjectURL(self.mediaSource);\n    self.mediaSource.addEventListener('sourceopen',\nself.onMediaSourceOpen.bind(self));\n\n};\n\nApp.prototype.onMediaSourceOpen = function () {\n    var self = this;\n\n    self.mediaSource.duration = (NUM_SEGMENTS - 1) * 2;\n    self.sourceBuffer = self.mediaSource.addSourceBuffer('video\/mp4; codecs=\"avc1.4d4028\"');\n    self.sourceBuffer.addEventListener('updateend', function () {\n        self.printBufferRanges();\n        if (ENABLE_GAP_WORKAROUND && self.currentSegment === 2 && !self.hasRemoved) {\n            self.remove(0, 2000);\n            self.hasRemoved = true;\n            return;\n        } \n\n     self.append();\n});\n\nApp.prototype.append = function () {\n    var self = this;\n\n    if (self.currentSegment >= content.length - 1) {\n        self.finished = true;\n    } else {\n        self.fetchSegment(content[self.currentSegment], function (arrayBuffer) {\n            try {\n                self.sourceBuffer.appendBuffer(arrayBuffer);\n                self.currentSegment += 1;\n            } catch (e) {\n                console.log(e);\n            }\n        });\n    }\n};\n\nApp.prototype.remove = function (start, end) {\n    var self = this;\n    self.debug('Removing buffer from ' + start + ' to ' + end)\n    self.sourceBuffer.remove(start, end);\n}\n<\/code><\/pre>\n\n\n\n
    Test Content<\/h3>\n\n\n\n
    Next, we need to create test content that we can feed into our MSE player. Based on Matt\u2019s comment above, we require two things: First, a media segment with a single IDR frame (IDR-segment) and a \u201chuge duration\u201d. Second, our \u201cdefault\u201d media content with a gap in the media buffer.<\/p>\n\n\n\n
    For that reason, we create a media segment with a single IDR frame and a duration of four seconds. That will be our \u201csimple keyframe with a huge duration\u201d. And to keep us motivated throughout the entire process we use our favorite bunny (the name is avoided here to not cause any mental issues with my fellow developer colleagues) as an input: There is no better feeling than replacing the bunny with something different.<\/p>\n\n\n\n
    \"\"<\/figure>\n\n\n\n
    The \u201cdefault\u201d content that we use to replace the IDR-segment with is encoded and packaged with<\/p>\n\n\n\n