Native Body Charge Playback. This text talks a couple of novel HDMI… | by Netflix Expertise Weblog | Jun, 2023
Maximizing immersion for our members is a crucial objective for the Netflix product and engineering groups to maintain our members entertained and absolutely engaged in our content material. Leveraging an excellent mixture of mature and cutting-edge shopper gadget applied sciences to ship a easy playback expertise with glitch-free in-app transitions is a crucial step in direction of reaching this objective. On this article we clarify our journey in direction of productizing a greater viewing expertise for our members by using options and capabilities in client streaming units.
When you have a streaming gadget related to your TV, resembling a Roku Set High Field (STB) or an Amazon FireTV Stick, you might have come throughout an possibility within the gadget show setting pertaining to content material body price. Gadget producers usually name this characteristic “Match Content material Body Charge”, “Auto alter show refresh price” or one thing comparable. In the event you’ve ever puzzled what these options are and the way they will enhance your viewing expertise, maintain studying — the next sections cowl the fundamentals of this characteristic and clarify the small print of how the Netflix utility makes use of it.
Netflix’s content material catalog consists of video captured and encoded in certainly one of varied body charges starting from 23.97 to 60 frames per second (fps). When a member chooses to look at a film or a TV present on a supply gadget (ex. Set-top field, Streaming stick, Recreation Console, and so forth…) the content material is delivered after which decoded at its native body price, which is the body price it was captured and encoded in. After the decode step, the supply gadget converts it to the HDMI output body price which was configured based mostly on the capabilities of the HDMI enter port of the related sink gadget (TV, AVR, Monitor and so forth). On the whole, the output body price over HDMI is routinely set to 50fps for PAL areas and 60fps for NTSC areas.
Netflix presents restricted excessive body price content material (50fps or 60fps), however the majority of our catalog and viewing hours may be attributed to members watching 23.97 to 30fps content material. This basically implies that more often than not, our content material goes via a course of referred to as body price conversion (aka FRC) on the supply gadget which converts the content material from its native body price to match the HDMI output body price by replicating frames. Determine 1 illustrates a easy FRC algorithm that converts 24fps content material to 60fps.
Changing the content material and transmitting it over HDMI on the output body price sounds logical and simple. The truth is, FRC works properly when the output body price is an integer a number of of the native body price ( ex. 24→48, 25→50, 30→60, 24→120, and so forth…). Then again, FRC introduces a visible artifact referred to as Judder when non-integer a number of conversion is required (ex. 24→60, 25→60, and so forth…), which manifests as uneven video playback as illustrated beneath:
You will need to observe that the severity of the judder is determined by the replication sample. Because of this, judder is extra distinguished in PAL areas due to the method of changing 24fps content material to 50fps over HDMI (see Determine 2):
- Whole of fifty frames should be transmitted over HDMI per second
- Supply gadget should replicate the unique 24 frames to fill within the lacking 26 frames
- 50 output frames from 24 authentic frames are derived as follows:
- 22 frames are duplicated ( whole of 44 frames )
- 2 frames are repeated 3 times ( whole of 6 frames )
As a evaluate, judder is extra pronounced when the frequency of the variety of repeated frames is inconsistent and unfold out e.g. within the situation talked about above, the body replication issue varies between 2 and three leading to a extra distinguished judder.
Now that now we have a greater understanding of the problem, let’s evaluate the options that Netflix has invested in. As a result of fragmented nature of gadget capabilities within the ecosystem, we explored a number of options to deal with this problem for as many units as doable. Every distinctive answer leverages present or new supply gadget capabilities and comes with varied tradeoffs.
The primary answer we explored and not too long ago enabled leverages the aptitude of present supply & sink units to vary the outgoing body price on the HDMI hyperlink. As soon as this characteristic is enabled within the system settings, units will match the HDMI output body price with the content material body price, both precisely or an integer a number of, with out consumer intervention.
Whereas this feels like the proper answer, units that help older HDMI applied sciences e.g. HDMI v<2.1, can’t change the body price with out additionally altering the HDMI knowledge price. This ends in what is usually referred as an “HDMI bonk” which causes the TV to show a clean display screen momentarily. Not solely is that this a disruptive expertise for members, however the period of the clean display screen varies relying on how briskly the supply and sink units can resynchronize. Determine 3 beneath is an instance of how this transition appears to be like:
Enhancements within the latest HDMI requirements (HDMI 2.1+) now enable a supply gadget to ship the video content material at its native body price with no need an HDMI resynchronization. That is doable via an progressive know-how referred to as Quick Media Switching (QMS) which is an extension of Variable Refresh Rate (VRR) focused for content material playback eventualities. QMS permits a supply gadget to keep up a relentless knowledge price on the HDMI hyperlink even throughout transmission of content material with completely different body charges. It does so by adjusting the quantity of non-visible padding knowledge whereas protecting the quantity of seen video knowledge fixed. As a result of fixed HDMI knowledge price, the HDMI transmitter and receiver don’t must resynchronize, resulting in a seamless/glitch-free transition as illustrated in Determine 4.
HDMI QMS is positioned to be the perfect answer to deal with the issue we’re presenting. Sadly, at current, this know-how is comparatively new and adoption into supply and sink units will take time.
Aside from the above HDMI specification dependent options, it’s doable for an utility like Netflix to control the presentation time stamp worth of every video body to attenuate the impact of judder i.e. the applying can current video frames to the underlying supply gadget platform at a cadence that may assist the supply gadget to attenuate the judder related to FRC on the HDMI output hyperlink.
Allow us to perceive this concept with the assistance of an instance. Let’s return to the identical 24 to 50 fps FRC situation that was coated earlier. However, as a substitute of excited about the FRC price per second (24 ⇒ 50 fps), let’s increase the FRC calculation time interval to three seconds (24*3 = 72 ⇒50*3 = 150 fps). For content material with a local body price of 24 fps, the supply gadget must get 72 frames from the streaming utility in a interval of three seconds. Now as a substitute of sending 24 frames per second at a daily per second cadence, for every 3 second interval the Netflix utility can resolve to ship 25 frames within the first 2 seconds (25 x 2 = 50) and 22 frames within the third second thereby nonetheless sending a complete of 72 (50+22) frames in 3 seconds. This strategy creates a fair FRC within the first 2 seconds (25 frames replicated twice evenly) and within the third second the supply gadget can do a 22 to 50 fps FRC which is able to create much less visible judder in comparison with the 24->50 fps FRC given a extra even body replication sample. This idea is illustrated in Determine 5 beneath.
NOTE: This answer was developed by David Zheng within the Companion Expertise Expertise crew at Netflix. Be careful for an upcoming article going into additional particulars of this answer.
Given the doable options out there to make use of and the related advantages and limitations, the Netflix utility working on a supply gadget adapts to make use of certainly one of these approaches based mostly on components resembling supply and sink gadget capabilities, consumer preferences and the precise use case throughout the Netflix utility. Let’s stroll via every of those facets briefly.
Each supply gadget that integrates the Netflix utility is required to let the applying know if it and the related sink gadget have the power to ship and obtain video content material at its native body price. As well as, a supply gadget is required to tell whether or not it may possibly help QMS and carry out a seamless playback begin of any content material at its native body price on the related HDMI hyperlink.
As mentioned within the introduction part, the presence of a system setting like “Match Content material Body Charge” sometimes signifies {that a} supply gadget is able to this characteristic.
Even when a supply gadget and the related sink can help Native content material body price streaming (seamless or non-seamless), a consumer may need chosen not to do that by way of the supply gadget system settings e.g. “Match Content material Body Charge” set to “By no means”. Or they could have indicated a choice of doing this solely when the native content material body price play begin can occur in a seamless method e.g. “Match Content material Body Charge” set to “Seamless”.
The Netflix utility must know this consumer choice so as to honor their choice. Therefore, supply units are anticipated to relay this consumer choice to the Netflix utility to assist with this run-time resolution making.
Despite supply gadget functionality and the consumer preferences collectively indicating that the Native Content material Body Charge streaming must be enabled, the Netflix utility can resolve to disable this characteristic for particular member experiences. For example, when the consumer is looking Netflix content material within the house UI, we can not play Netflix trailers of their Native body price as a result of following causes:
- If utilizing Answer # 1, when the Netflix trailers are encoded in various content material body charges, switching between trailers will lead to display screen blanking, thereby making the UI looking unusable.
- If utilizing Answer # 2, sending Netflix trailers of their Native body price would imply that the related UI parts (motion of cursor, asset choice and so forth) would even be displayed on the decreased body price and this may lead to a sluggish UI looking expertise. It’s because on HDMI output from the supply gadget, each graphics (Netflix utility UI) and video parts will exit on the identical body price (native content material body price of the trailer) after being blended collectively on the supply gadget.
To deal with these points we comply with an strategy as proven in Determine 6 beneath the place we allow the Native Body Charge playback expertise solely when the consumer selects a title and watches it in full display screen with minimal graphical UI components.
This text introduced options that purpose to enhance the content material playback expertise on HDMI supply units. The breadth of accessible technical options, consumer selectable preferences, gadget capabilities and the applying of every of those permutations within the context of assorted in-app member journeys characterize a typical engineering and product resolution framework at Netflix. Right here at Netflix, our objective is to maximise immersion for our members via introduction of latest options that may enhance their viewing expertise and maintain them absolutely engaged in our content material.
We want to acknowledge the onerous work of quite a few groups that got here collectively to ship the options being mentioned on this doc. These embody Core UI and JS Participant improvement, Netflix Utility Software program improvement, AV Take a look at and Tooling (earlier article from this crew), Companion Engineering and Product groups within the Consumer Engineering group and our knowledge science associates within the Data Science and Engineering group at Netflix. Diagrams on this article are courtesy of our Companion Enterprise Platform XD crew.
