Solving ExoPlayer rendering performance issues

In a recent blog post by Google Exoplayer developers new experimental features to improve the ExoPlayer’s rendering performance were introduced. As we have received numerous customer complaints about stuttering, freezing and frame drops during video playback on various Android-based devices, we were excited to have a closer look at the new ExoPlayer decoding modes.

Test Setup

The following list describes the supported media codec operation modes, which are available in the dev-v2 branch of the ExoPlayer GitHub project. The repository at this point in history was used:

For the purpose of this blog post, we are anonymizing Android phones and tablets manufacturer and model names:

#DeviceAndroid (SDK) version
1Android Phone10 (29)
2Android Phone9 (28)
3Android Phone9 (28)
4Android Tablet9 (28)
Test devices. The cryptic numbers are the Android and SDK version.

Last thing we needed before we could start experimenting were some test streams. We chose MPEG-DASH live streams with different encoding ladders and encoded the representations with H.264 high profile (some with 50 fps), since this combination causes the biggest performance issues. The streams #1 and #8 were encrypted and the stream #7 was unencrypted. Here are the stream configurations:

#ResolutionBitratefpsCodecEncrypted
11280×7208.910.00050,00avc1.640020
1280×7205.038.00050,00avc1.640020
960×5402.655.00050,00avc1.64001f
480×270673.00025,00avc1.640015
71280×7208.000.00050,00avc1.64001f
960×5401.500.00050,00avc1.64001f
640×360600.00050,00avc1.64001e
81920×10808.741.00050,00avc1.64002a
1280×7204.869.00050,00avc1.640020
960×5402.486.00050,00avc1.64001f
640×3601.245.00025,00avc1.64001e
480×270590.00025,00avc1.640015
DASH live stream configuration

For our experiments the following KPIs are observed:

  • dk – (droppedToKeyframeCount) The number of times all buffers to a keyframe were dropped. Each time buffers to a keyframe are dropped, this counter is increased by one, and the dropped buffer counters are increased by one (for the current output buffer) plus the number of buffers dropped from the source to advance to the keyframe
  • vfpo – video frame processing offset
    • value how early or late a frame is processed regarding presentation time
      • e.g. a frame is 30ms early => vfpo = 30.000
      • e.g. a frame is 10ms late => vfpo = -10.000
    • smooth playback with an average vfpo of 40.000

Evaluation

We have found that due to the correlation between the dropped keyframes and the video frame processing offset (vfpo), the 5 experimental modes can lead to a reduction of the dropped keyframes or an increase of the video frame processing offset.

The following diagrams show the change of the video frame processing offset (vfpo) on one device at a time, the same encrypted stream #1 under the different modes. Horizontally from left to right the parameters for modes 0 to 5 are shown. Note that only the highest representation of the stream is shown in the following diagrams.

The blue curve shows the video frame processing offset (vfpo) after a playback time of at least 60 seconds. A positive value around 40.000 average vfpo corresponds to a smooth playback. Negative values for the vfpo subjectively show micro-stuttering.

Regarding the correlation of micro-stuttering to dropped keyframes the orange line shows the number of dropped keyframes in the respective mode.

On Device #2 the video frame processing offset (vfpo) value remains around 40,000 in each mode. The benchmarking for this device shows that there are no stuttering issues.

For Device #1 and Device #3 there is a numerical improvement in the video frame processing offset (vfpo). While this value improves positively on Device #3 and leads to a smooth playback in modes 4 and 5, the improved vfpo values on Device #1 and Device #4 remain in the negative range and continue to show micro stuttering (for Device #4 nearly smooth playback but micro stuttering in Mode 4 and 5).

It is interesting to note that the modes on the tested devices show a varying degree of change in the vfpo – especially Device #4 shows an improvement of more than 700.000 with modes 4 and 5, the average improvement is in the range of about 300.000.

For a comparison between encrypted and unencrypted streams, the tests were also conducted with unencrypted streams. The following diagrams for the Devices #1 and #3 show the change of the video frame processing offset (vfpo) on one device at a time and the same unencrypted stream #7 under the different modes.

The playback of unencrypted streams is more resource efficient; this can be seen in the corresponding video frame processing offset (vfpo) – with values remaining around 40,000 – and dropped keyframes metrics which indicate smooth playback in each mode.

The tests were repeated with stream #8 which is encrypted and has a resolution of FullHD in the highest representation. For comparison, the diagrams for the Devices #1 and #3 are shown below:

Again for Device #1 and Device #3 there is a numerical improvement in the video frame processing offset (vfpo). While this value similarly to stream #1 improves positively on Device #3 and leads to a smooth playback in modes 4 and 5, the improved vfpo values on Device #1 remain in the negative range and continue to show micro stuttering.

It can be observed that the efficiency of resources, e.g. in the direct comparison between encrypted and unencrypted streams with a similar representation and streams with a similar video codec profile at higher representation (1080p vs. 720p), results in diagrams with a similar pattern. particularly noticeable is the comparison of the two diagrams for device #1 for streams #1 and #8, which at the higher representation looks like a parallelshift of the curve into the negative range.

Summary

The first findings indicate that modes 4 and 5 for the experimental MediaCodecOperationMode improve the video playback performance with regard to the video frame processing offset (vfpo). We have integrated vfpo as an additional streaming metric into our streaming analytics solution SAND.

While the new modes do not completely solve video playback performance issues on all tested devices, they always perform better than mode 0. Therefore, we recommend to use the new modes, especially mode 4 and 5, if you are suffering from stuttering, freezing and similar playback performance-related issues on Android-based devices running Exoplayer.

Please note that these experimental modes are not available in a stable ExoPlayer release yet. According to Google Exoplayer developers they will be integrated in one of the upcoming releases. We will conduct additional tests once this happens.

If you have any additional question regarding our SAND or DASH activities, feel free to comment below or contact us directly.

Leave a Reply

Your email address will not be published.