Ian Goodfellow: Machine learning researcher who coined generative adversarial networks (GANs) in his 2014 paper, co-author of the textbook Deep Learning, and director of machine learning at Apple (formerly OpenAI and Google Brain).
What this episode covers
Lex Fridman interviews Ian Goodfellow about the current limits of deep learning, including its heavy reliance on labeled data and its relationship to reasoning and cognition. Goodfellow explains how generative adversarial networks work as a two-player game between a generator and a discriminator that reaches a Nash equilibrium. They discuss adversarial examples as a security liability, semi-supervised learning, fairness, differential privacy, and using GANs for data augmentation. The conversation closes on deepfakes, authentication, what artificial general intelligence might require, and the importance of securing machine learning against adversaries.
The guest's own work
Deep Learning
Ian Goodfellow, Yoshua Bengio, Aaron Courville
“he's the author of the popular textbook on deep learning simply titled deep learning”— Lex Fridman
Also referenced (named, not recommended)
Artificial Intelligence: A Modern Approach
Stuart Russell, Peter Norvig
“you're writing the deep learning chapter for the fourth edition of the artificial intelligence the modern approach book”— Lex Fridman
Big reveals from this episode
- Goodfellow says the GAN idea came from an argument at a bar; he went home at midnight, coded it, and it worked the first time.
- He admits a little drinking lowered his inhibitions enough to try the idea, which he might have shot down sober.
- He explains the generator/discriminator game where the generator learns to produce realistic data by fooling the discriminator.
- He reveals generative models that did exactly what is asked would just memorize the training data, yet GANs generalize anyway and nobody fully knows why.
- A semi-supervised GAN got below 1% MNIST error with only 100 labeled examples, a roughly 600x reduction in needed labels.
- Goodfellow says he is far less worried about deepfakes 20 years out than the next few bumpy years of cultural transition.
- He says it is straightforward to prove he is human today, but proving content is real from the content alone is incredibly hard.
Worth remembering
- His deep learning book opens with a Russian-doll diagram showing deep learning inside representation learning inside machine learning inside AI.
- Humans do not learn to play Pong by failing two million times, unlike reinforcement learning algorithms.
- A 2016 paper called Hidden Voice Commands made sounds humans cannot understand but phones recognize as target commands.
- The Deep Image Prior paper shows a convolutional net architecture is useful for inpainting even without learning any parameters.
- The first GAN paper used MNIST, the Toronto Face Database, and CIFAR-10, with mostly unrecognizable early samples.
- The DC-GAN paper sparked a Cambrian explosion of GAN variants and became a widely used baseline backbone.
- CycleGAN tends to make horses greener because horses appear on grass while zebras appear on drier terrain in the data.
- A paper from Casey Greene's lab trained GANs with differential privacy to produce fake patient data safe for researchers.