Yann Lecun: Meta AI, Open Source, Limits of LLMs, AGI & the Future of AI | Lex Fridman Podcast #416

Yann LeCun explains why he believes autoregressive large language models like GPT-4 and Llama are missing essential components of intelligence and won't lead to human-level AI. He lays out his alternative vision built on joint embedding predictive architectures (JEPA), world models learned from video, energy-based models, and objective-driven planning. The conversation covers hallucinations, the Moravec paradox, and why sensory experience carries vastly more information than language. LeCun makes a forceful case for open-source AI as the only path to diverse, democratic AI systems and pushes back hard against AI doomers, arguing AGI will arrive gradually with guardrails rather than as a single catastrophic event.

• LeCun argues autoregressive LLMs are not the path to superhuman intelligence because they lack four essential traits: understanding the physical world, persistent memory, reasoning, and planning.
  00:02:34
• A four-year-old has absorbed roughly 10^15 bytes through vision versus only ~2x10^13 bytes in all the public text an LLM trains on, showing sensory input dwarfs language as a learning source.
  00:04:39
• Training systems to reconstruct corrupted images self-supervised produces poor representations; LeCun's alternative is joint embedding (JEPA), predicting abstract representations rather than pixels.
  00:24:37
• Hallucinations are inherent: because of autoregressive prediction, the probability of staying in the set of correct answers decreases exponentially with the number of tokens produced.
  01:06:26
• LeCun's summary recommendation: abandon generative models, autoregressive generation, probabilistic models, and contrastive methods in favor of JEPA, energy-based models, and regularized methods.
  01:28:47
• He warns we cannot afford AI systems mediating all human knowledge to come from a handful of West Coast companies; open source is the answer to preserve diversity and democracy.
  01:38:48
• He calls AI doom scenarios mostly false: super-intelligence won't be a sudden event, intelligent systems won't inherently want to dominate, and the desire to dominate must be hardwired, not emergent.
  02:08:57
• LeCun's hope for humanity: AI will amplify human intelligence so everyone effectively manages a staff of smart AI assistants, like the printing press making everyone smarter.
  02:38:14

• LLMs train on roughly 10^13 tokens of public text; it would take a human 170,000 years reading 8 hours a day to get through that much data.
  00:04:07
• The Moravec paradox: computers easily play chess and solve integrals but struggle with tasks like driving or clearing a dinner table that any child does effortlessly.
  00:08:50
• Meta developed V-JEPA, applying the JEPA idea to video by masking a spatiotemporal 'tube' (typically 16 frames) to learn good video representations.
  00:38:42
• V-JEPA representations can preliminarily tell whether a video is physically possible or impossible based on whether objects appear, disappear, or jump location.
  00:39:45
• Meta has speech-to-speech translation systems covering hundreds of languages, including languages that have no written form, going directly speech-to-speech.
  00:58:08
• The founder of Infosys is funding a project to fine-tune Llama 2 to speak all 22 official languages of India.
  01:40:52
• Studies suggest LLMs don't meaningfully help someone build a bioweapon beyond what a search engine and library already provide, and actual fabrication requires hard-to-acquire physical expertise.
  01:55:26
• A GPU consumes about half a kilowatt to a kilowatt while the human brain runs on about 25 watts, leaving AI off by a factor of roughly 100,000 to a million in power efficiency.
  02:03:48
• The Ottoman Empire banned the printing press for Arabic for 200 years, partly to protect the powerful corporation of calligraphers and scribes.
  02:42:51
• LeCun cites Gemini generating images of a black George Washington and refusing to depict Tiananmen Square's Tank Man as examples of debiasing gone wrong.
  01:34:36