Lex Fridman: AI researcher and host of the Lex Fridman Podcast. Here he presents solo, kicking off an AI paper reading club with Turing's foundational paper.
What this episode covers
This is a solo lecture-style presentation, the first in Lex Fridman's AI paper reading club, walking through Alan Turing's 1950 paper 'Computing Machinery and Intelligence' and the imitation game it proposes. Lex explains the Turing test, its real-world implementations like the Loebner Prize and the Eugene Goostman claim, and Google's Meena chatbot with its sensibleness/specificity metric. He methodically covers Turing's nine objections plus Searle's Chinese Room, then surveys alternative tests including the Lovelace test, Winograd Schema Challenge, Amazon Alexa Prize, Hutter Prize compression challenge, and Francois Chollet's ARC benchmark. He concludes that natural-language conversation remains the ultimate test of intelligence and that researchers wrongly dismiss it as a distraction.
Recommended on this episode
On the Measure of Intelligence
Francois Chollet
“here's just a couple of example of priors that Francois shows in his paper I recommend highly it called on the measure of intelligence”— Lex Fridman
Also referenced (named, not recommended)
Computing Machinery and Intelligence
Alan Turing
“the title of the paper was Computing Machinery and intelligence published almost 70 years ago in 1950 author Alan Turing”— Lex Fridman
Minds, Brains and Programs
John Searle
“the most famous objection to the Turing test proposed by John Searle in 1980 in his paper minds brains and programs”— Lex Fridman
Nitro Cold Brew
Starbucks
“this video is brought to you by this magic potion called nitro cold brew an excessively expensive canned beverage from Starbucks that fuels me”— Lex Fridman
Alexa
Amazon
“they can use a I think it's called a social bot skill on there Alexa devices and I don't want to wake up my own Alexa devices”— Lex Fridman
Big reveals from this episode
- Lex notes the Loebner Prize is apparently no longer funded, lamenting that major labs like DeepMind and Facebook AI never took on the Turing test challenge.
- He argues nobody has researched how to make explicit exactly which parts of a conversation reveal a bot is not human.
- He calls out Google's Meena results as possibly PR-driven and closed-source, urging the numbers be taken with a grain of salt.
- Lex says he goes back and forth daily on whether mimicking thinking equals thinking, mimicking consciousness equals consciousness.
- He admits the poetic claim 'the appearance of consciousness is consciousness' reflects his real engineering stance.
- He openly disagrees with Francois Chollet and Stuart Russell, insisting the Turing test is not a distraction but keeps researchers honest.
- He concludes the real test of human-level intelligence will be deep, meaningful connection between human and machine via open-domain conversation.
Worth remembering
- Turing predicted that by 2000 a machine with 100MB of storage would fool 30% of humans in a five-minute conversation.
- The Loebner Prize, running since 1991, offered $25,000 for text-only passing and $100,000 for multimodal.
- In 2014 the Eugene Goostman bot fooled 33% of judges by posing as a 13-year-old Ukrainian boy with a language barrier.
- Humans score 97% on sensibleness; on combined sensibleness and specificity humans hit 86%, Meena 79%, Mitsuku 56%.
- The Lovelace objection from Ada Lovelace holds machines can only do what we program; Turing rephrased it as 'machines can't surprise us.'
- The Hutter Prize compresses 1GB of Wikipedia, current best is ~117MB, paying 5,000 euros per 1% improvement.
- Lex muses that intelligence may be the journey, not the destination, judging systems over time rather than instantaneously.
- Francois Chollet's ARC challenge uses a colored grid world to test abstract reasoning close to human IQ tests.