FFmpeg: The Incredible Technology Behind Video on the Internet | Lex Fridman Podcast #496

Lex Fridman talks with FFmpeg and VLC developers Jean-Baptiste Kempf and Kieran Kunhya about the open-source software that quietly powers almost all video and audio on the internet, from YouTube and Netflix to Chrome and Discord. They walk through how video playback works end-to-end (containers, codecs, demuxing, decoding), why codecs degrade signals to match human perception, and why handwritten assembly massively outperforms modern compilers. The conversation covers the open-source movement, licensing, maintainer burnout, the Google AI security-report debacle, and the under-appreciated volunteers who keep critical infrastructure running. They also dig into the history of x264, H.264, AV1/AV2, dav1d, reverse engineering of proprietary codecs, the archiving community, and the future of multimedia including 3D, haptics, and brain-computer interfaces. Throughout, JB recounts refusing tens of millions of dollars to keep VLC ad-free and the philosophy of building excellent tools for the greater good.

• Intelligence agencies asked twice if a backdoor could be put in VLC, and the developers refused, saying they would shut the software down rather than compromise it.
  00:01:32
• JB repeatedly turned down tens of millions of dollars (the last offer described as obscene) to keep VLC open source, ad-free, and tracker-free.
  00:55:54
• dav1d, VideoLAN's AV1 decoder, is roughly 30,000 lines of C and 240,000 lines of handwritten assembly, possibly the largest assembly codebase, optimized so 'every cycle matters.'
  02:07:28
• Google used AI to generate security reports on FFmpeg, publicly promoting its AI before volunteers could fix issues, marking even an obscure 1990s game-codec bug as high priority.
  01:10:48
• WikiLeaks' Vault 7 revealed the CIA used a modified version of VLC with an added DLL (psapi.dll) to read, encrypt, and exfiltrate users' documents while they watched a movie.
  03:10:57
• JB received a death threat with powder in it (during the anthrax-scare era) after deciding to stop maintaining VLC's PowerPC port around 2009-2010.
  02:44:06
• JB's secret to staying zen: he constantly asks 'in the end, am I dead? Am I hurting someone?', and notes the source code is out there and unstoppable.
  04:16:14
• VideoLAN has no office and spends less than ten thousand dollars per year on lawyers; VLC runs entirely client-side with no telemetry, so it cannot surveil or censor what users watch.
  03:51:55

• Around 30 percent of Netflix video and 50 percent of YouTube is now in AV1, running on an estimated three billion devices decoding video nonstop.
  00:00:46
• About 25 percent of traffic to VLC's main website comes from people googling 'cone player' rather than VLC.
  00:07:32
• Video compression is roughly 100x to 200x (versus about 10x for MP3 audio), and all codecs are designed around human ear and eye perception, working in YUV rather than RGB.
  00:14:33
• Re-licensing VLC's core from GPL to LGPL required JB to track down and get agreement from more than 350 individual contributors, including a deceased contributor's father.
  00:48:10
• Linus Torvalds built Git in about two weeks, which JB argues is more interesting than the Linux kernel itself.
  01:35:23
• Reverse-engineering a one-megabyte binary blob takes roughly a month of work, yet developer Kostya Shishkov reverse-engineered 20-30 megabyte blobs of obscure codecs for fun.
  01:44:56
• Handwritten SIMD assembly in FFmpeg/dav1d delivers 10x to 50x (and in one example 62x) speedups over C, with developers sometimes abusing cryptography instructions for unrelated video tasks.
  02:05:54
• dav1d uses runtime processor detection and even bypasses the operating system's standard calling convention for speed, with separate handwritten assembly for every instruction set (x86, ARM64, RISC-V, etc.).
  02:13:55
• FFmpeg runs on NASA's Mars 2020 rover to compress pictures, making it, as Kieran jokes, a multi-planetary open-source library.
  04:14:10
• The 'ParkJoy' test sequence, shot on film by Swedish television and given away free, became the canonical benchmark that exposed how high-PSNR encoders blur detail in grass, water, and trees.
  02:53:15