Cristos Goodrow: VP of Engineering at Google who leads the team responsible for YouTube's search and recommendation systems. He has worked on YouTube's machine learning ranking and quality measurement for many years.
What this episode covers
Lex Fridman interviews Cristos Goodrow about the foundations of the YouTube recommendation and search systems. Goodrow explains collaborative filtering and how watching videos together creates a 'related graph' that naturally clusters videos by language and topic. They discuss the evolution of quality signals from views, to watch time, to post-watch satisfaction surveys, plus likes, shares, subscribes, and dismissals. Goodrow also covers clickbait, gaming the system, A/B testing with hundreds of variables, and the idea that the 'algorithm' is really code married with the collective behavior of all viewers.
Recommended on this episode
How the Economic Machine Works
Ray Dalio
“Ray Dalio has this video on the economic machine I forget what it's called but it's a 30-minute video and I said it's the greatest video I've ever watched”— Lex Fridman
Also referenced (named, not recommended)
Classical Recursion Theory
Piergiorgio Odifreddi
“the people who who write classical recursion theory by pure OD Freddie it was fine with the yellow title and the and nothing more”— Cristos Goodrow
World of Warcraft
Blizzard Entertainment
“we were a few years ago having a live stream of some competition for world of warcraft on YouTube and it was a very important competition”— Cristos Goodrow
Minecraft
Mojang
“what is the highest quality prank video for what is the highest quality minecraft video yeah right that might be the one that people enjoy watching the most”— Cristos Goodrow
Big reveals from this episode
- The first real attempt at 'watch next' recommendations used collaborative filtering, observing which videos get watched close together by the same person.
- The resulting 'related graph' automatically groups videos by language and topic without anyone programming language detection.
- YouTube moved from optimizing for views to watch time because views can be inflated by misleading clicks people quickly abandon.
- YouTube surveys users the next day, asking whether they'd rate a video four or five stars, and feeds that into the ML system.
- Content analysis of video itself is still 'somewhat crude' - it can tell soccer is being played but not whether it's Manchester United.
- What outsiders call 'the YouTube algorithm' is really code plus the live behavior of every viewer; without people the algorithm wouldn't work.
- An early diversity rule was a simple heuristic: no more than three videos in a row from the same channel.
Worth remembering
- A researcher couldn't believe YouTube recommended English academic videos but Turkish baklava videos, matching her bilingual habits automatically.
- YouTube sees each user essentially as the full history of videos they've watched, like a 'DNA strand' or a vector in video space.
- Some users don't subscribe because they think it costs money; others subscribe just to give a creator a 'high five' without watching more.
- A major World of Warcraft livestream was hard to find because the title was just 'match 478 - A team versus B team,' not the game name.
- If thumbnails are too racy or have too many exclamation points, YouTube suppresses them because users report being offended.
- Nearly every change to YouTube is shipped only after an A/B experiment running one week to months, measuring literally hundreds of variables.
- The simplest success signal is whether a user returns on another day - people don't come back to things they don't value.
- Some people and machines deliberately try to associate unrelated videos together because it's profitable, so systems must resist gaming.