Dawn Song: Dawn Song is a professor of computer science at UC Berkeley specializing in computer security, adversarial machine learning, and privacy. She is also founder of the startup Oasis Labs.
What this episode covers
Lex Fridman talks with UC Berkeley professor Dawn Song about the intersection of computer security and machine learning. They cover formally verified systems, how attackers fool neural networks with adversarial examples in both the digital and physical world (including stop signs and facial recognition), and how training data can be poisoned or extracted. The conversation expands into data privacy, differential privacy, data ownership as an economic right, blockchain security, and program synthesis. It closes with a reflective discussion of Song's journey from physics in China to computer science in the US and the meaning of life.
Also referenced (named, not recommended)
Google Translate
“my students and collaborators has shown some very effective attacks on real-world systems for example Google Translate and translation api's”— Dawn Song
Tesla Autopilot
Tesla
“in the context of autonomous vehicles that use like Tesla autopilot for example they uses vision as a primary sensor for perceiving the world”— Lex Fridman
Tetris
Alexey Pajitnov
“the other one actually programmed the game that's like a 3d Tetris it was a to not to be a super hard game to play”— Dawn Song
Big reveals from this episode
- Song argues security attacks are moving up the stack toward humans, who are the weakest link and cannot be patched like software.
- Her team showed you can poison facial recognition so that anyone wearing a specific pair of glasses gets recognized as a chosen target like a president.
- Adversarial stop signs her team created are robust enough to fool autonomous-driving classifiers and are now exhibited at a London science museum.
- Her group stole a working imitation of Google Translate via API queries, then crafted attacks that transferred to the real system, flipping '6 Fahrenheit' to '21'.
- By querying a language model trained on the Enron email dataset, attackers extracted real Social Security and credit card numbers from the training data.
- Song says she is '100 percent confident' physical adversarial attacks on Tesla are feasible, disagreeing with Elon Musk who dismissed the concern.
- She frames data ownership as a property right that, like physical property rights historically, could be a major driver of economic growth.
Worth remembering
- Formally verified systems now exist ranging from microkernels to compilers to file systems to crypto libraries, with teams spending decades on them.
- Song cites the saying 'security is job security' because you can never prove a real-world system has zero vulnerabilities.
- Backdoor data-poisoning attacks can be so stealthy that humans reviewing the training set cannot see them and the model behaves correctly except on trigger inputs.
- Her spatial-consistency defense detects adversarial attacks on image segmentation because overlapping patches must agree at their intersection.
- A differentially private language model preserves utility while defeating attacks that extract sensitive data from a vanilla model.
- On public blockchains nothing is private by default; confidentiality requires extra techniques like zero-knowledge proofs and secure computation.
- Song started in a physics PhD at Cornell before switching to computer science at Carnegie Mellon.
- Song concludes that you yourself must define the meaning of your life, which is both a burden and a freedom.