As with other codecs, service providers using VP9 need to think about bitrate - the number of ones and zeros required to send each frame of a video. Our initial focus is on the VP9 codec (specifically the open source version libvpx), since it’s widely used by YouTube and other streaming services. However, because RL is particularly well-suited to sequential decision-making problems like those in codecs, we’re exploring how an RL-learned algorithm can help. Decades of hand engineering have gone into optimising these codecs, which are responsible for many of the video experiences now possible on the internet, including video on demand, video calls, video games, and virtual reality.

These codecs make multiple decisions for each frame in a video. Most online videos rely on a program called a codec to compress or encode the video at its source, transmit it over the internet to the viewer, and then decompress or decode it for playback. Since launching to production on a portion of YouTube’s live traffic, we’ve demonstrated an average 4% bitrate reduction across a large, diverse set of videos. With video surging during the COVID-19 pandemic and the total amount of internet traffic expected to grow in the future, video compression is an increasingly important problem - and a natural area to apply Reinforcement Learning (RL) to improve upon the state of the art in a challenging domain. Analysts predicted that streaming video will have accounted for the vast majority of internet traffic in 2021. In a preprint published on arXiv, we detail our collaboration with YouTube to explore the potential for MuZero to improve video compression. Now, in pursuit of DeepMind’s mission to solve intelligence, MuZero has taken a first step towards mastering a real-world task by optimising video on YouTube. But so far these agents have focused on solving games.

MuZero, for example, mastered Chess, Go, Shogi, and Atari without needing to be told the rules.

Its successors, AlphaZero and then MuZero, each represented a significant step forward in the pursuit of general-purpose algorithms, mastering a greater number of games with even less predefined knowledge. In 2016, we introduced AlphaGo, the first artificial intelligence program to defeat humans at the ancient game of Go. Collaborating with YouTube to optimise video compression in the open source VP9 codec.

Real control 2