In a recent blog post by Google Exoplayer developers new experimental features to improve the ExoPlayer’s rendering performance were introduced. As we have received numerous customer complaints about stuttering, freezing and frame drops during video playback on various Android-based devices, we were excited to have a closer look at the new ExoPlayer decoding modes.
Test Setup
The following list describes the supported media codec operation modes, which are available in the dev-v2 branch of the ExoPlayer GitHub project. The repository at this point in history was used:
- Mode 0: OPERATION MODE SYNCHRONOUS
- Maintains the current behavior. This is the default value.
- Mode 1: OPERATION_MODE_ASYNCHRONOUS_PLAYBACK_THREAD
- In this mode, ExoPlayer operates the MediaCodec in asynchronous mode and MediaCodec callbacks are routed to the player’s playback thread. This mode is applicable when API level ≥ 21. If API level < 21, the operation will fall back to OPERATION_MODE_SYNCHRONOUS.
- Mode 2: OPERATION MODE ASYNCHRONOUS DEDICATED THREAD
- In this mode, ExoPlayer operates the MediaCodec in asynchronous mode and callbacks are routed to a separate thread, one for each MediaCodec instance. This mode is applicable when API level ≥ 23. If API level < 23, the operation will fall back to OPERATION_MODE_SYNCHRONOUS.
- Mode 3: OPERATION MODE ASYNCHRONOUS DEDICATED THREAD MULTI LOCK
- Same as OPERATION_MODE_ASYNCHRONOUS_DEDICATED_THREAD, but the internal implementation is slightly optimized by performing more fine-grained locking on the internal data structures for input and output buffers.
- Mode 4: OPERATION MODE ASYNCHRONOUS DEDICATED THREAD ASYNCHRONOUS QUEUEING
- The mode extends OPERATION_MODE_ASYNCHRONOUS_DEDICATED_THREAD and offloads queueing input buffers to another thread.
- Mode 5: OPERATION MODE ASYNCHRONOUS DEDICATED THREAD MULTI LOCK ASYNCHRONOUS QUEUEING
- The mode extends OPERATION_MODE_ASYNCHRONOUS_DEDICATED_THREAD_MULTI_LOCK and offloads queueing input buffers to another thread.
For the purpose of this blog post, we are anonymizing Android phones and tablets manufacturer and model names:
| # | Device | Android (SDK) version |
| 1 | Android Phone | 10 (29) |
| 2 | Android Phone | 9 (28) |
| 3 | Android Phone | 9 (28) |
| 4 | Android Tablet | 9 (28) |
Last thing we needed before we could start experimenting were some test streams. We chose MPEG-DASH live streams with different encoding ladders and encoded the representations with H.264 high profile (some with 50 fps), since this combination causes the biggest performance issues. The streams #1 and #8 were encrypted and the stream #7 was unencrypted. Here are the stream configurations:
| # | Resolution | Bitrate | fps | Codec | Encrypted |
| 1 | 1280×720 | 8.910.000 | 50,00 | avc1.640020 | ✓ |
| 1280×720 | 5.038.000 | 50,00 | avc1.640020 | ✓ | |
| 960×540 | 2.655.000 | 50,00 | avc1.64001f | ✓ | |
| 480×270 | 673.000 | 25,00 | avc1.640015 | ✓ | |
| 7 | 1280×720 | 8.000.000 | 50,00 | avc1.64001f | ✗ |
| 960×540 | 1.500.000 | 50,00 | avc1.64001f | ✗ | |
| 640×360 | 600.000 | 50,00 | avc1.64001e | ✗ | |
| 8 | 1920×1080 | 8.741.000 | 50,00 | avc1.64002a | ✓ |
| 1280×720 | 4.869.000 | 50,00 | avc1.640020 | ✓ | |
| 960×540 | 2.486.000 | 50,00 | avc1.64001f | ✓ | |
| 640×360 | 1.245.000 | 25,00 | avc1.64001e | ✓ | |
| 480×270 | 590.000 | 25,00 | avc1.640015 | ✓ |
For our experiments the following KPIs are observed:
- dk – (droppedToKeyframeCount) The number of times all buffers to a keyframe were dropped. Each time buffers to a keyframe are dropped, this counter is increased by one, and the dropped buffer counters are increased by one (for the current output buffer) plus the number of buffers dropped from the source to advance to the keyframe
- vfpo – video frame processing offset
- value how early or late a frame is processed regarding presentation time
- e.g. a frame is 30ms early => vfpo = 30.000
- e.g. a frame is 10ms late => vfpo = -10.000
- smooth playback with an average vfpo of 40.000
- value how early or late a frame is processed regarding presentation time
Evaluation
We have found that due to the correlation between the dropped keyframes and the video frame processing offset (vfpo), the 5 experimental modes can lead to a reduction of the dropped keyframes or an increase of the video frame processing offset.
The following diagrams show the change of the video frame processing offset (vfpo) on one device at a time, the same encrypted stream #1 under the different modes. Horizontally from left to right the parameters for modes 0 to 5 are shown. Note that only the highest representation of the stream is shown in the following diagrams.
The blue curve shows the video frame processing offset (vfpo) after a playback time of at least 60 seconds. A positive value around 40.000 average vfpo corresponds to a smooth playback. Negative values for the vfpo subjectively show micro-stuttering.
Regarding the correlation of micro-stuttering to dropped keyframes the orange line shows the number of dropped keyframes in the respective mode.
On Device #2 the video frame processing offset (vfpo) value remains around 40,000 in each mode. The benchmarking for this device shows that there are no stuttering issues.
For Device #1 and Device #3 there is a numerical improvement in the video frame processing offset (vfpo). While this value improves positively on Device #3 and leads to a smooth playback in modes 4 and 5, the improved vfpo values on Device #1 and Device #4 remain in the negative range and continue to show micro stuttering (for Device #4 nearly smooth playback but micro stuttering in Mode 4 and 5).
It is interesting to note that the modes on the tested devices show a varying degree of change in the vfpo – especially Device #4 shows an improvement of more than 700.000 with modes 4 and 5, the average improvement is in the range of about 300.000.
For a comparison between encrypted and unencrypted streams, the tests were also conducted with unencrypted streams. The following diagrams for the Devices #1 and #3 show the change of the video frame processing offset (vfpo) on one device at a time and the same unencrypted stream #7 under the different modes.
The playback of unencrypted streams is more resource efficient; this can be seen in the corresponding video frame processing offset (vfpo) – with values remaining around 40,000 – and dropped keyframes metrics which indicate smooth playback in each mode.
The tests were repeated with stream #8 which is encrypted and has a resolution of FullHD in the highest representation. For comparison, the diagrams for the Devices #1 and #3 are shown below:
Again for Device #1 and Device #3 there is a numerical improvement in the video frame processing offset (vfpo). While this value similarly to stream #1 improves positively on Device #3 and leads to a smooth playback in modes 4 and 5, the improved vfpo values on Device #1 remain in the negative range and continue to show micro stuttering.
It can be observed that the efficiency of resources, e.g. in the direct comparison between encrypted and unencrypted streams with a similar representation and streams with a similar video codec profile at higher representation (1080p vs. 720p), results in diagrams with a similar pattern. particularly noticeable is the comparison of the two diagrams for device #1 for streams #1 and #8, which at the higher representation looks like a parallelshift of the curve into the negative range.
Summary
The first findings indicate that modes 4 and 5 for the experimental MediaCodecOperationMode improve the video playback performance with regard to the video frame processing offset (vfpo). We have integrated vfpo as an additional streaming metric into our streaming analytics solution SAND.
While the new modes do not completely solve video playback performance issues on all tested devices, they always perform better than mode 0. Therefore, we recommend to use the new modes, especially mode 4 and 5, if you are suffering from stuttering, freezing and similar playback performance-related issues on Android-based devices running Exoplayer.
Please note that these experimental modes are not available in a stable ExoPlayer release yet. According to Google Exoplayer developers they will be integrated in one of the upcoming releases. We will conduct additional tests once this happens.
If you have any additional question regarding our SAND or DASH activities, feel free to comment below or contact us directly.