Jeremy Howard: fast.ai Deep Learning Courses and Research | Lex Fridman Podcast #35

Jeremy Howard traces his path from programming on a Commodore 64 through esoteric array languages like APL and J to building fast.ai. He argues that most deep learning research is a waste of time and that the real impact comes from empowering domain experts with practical, accessible tools like transfer learning and active learning. He recounts how a handful of his students beat Google and Intel on Stanford's DAWNBench competition by training on cheap single-GPU setups using tricks like progressive resizing and super-convergence learning rates. He shares strong opinions on programming languages (Python is slow and unhackable, Swift is the hope, TensorFlow is a mess), and warns about labor force displacement and ethics in AI. He closes with his self-funded startup philosophy and his use of spaced repetition for learning Chinese.

• Jeremy's small team of about four students beat Google and Intel on the Stanford DAWNBench competition, topping both the time and cost leaderboards.
  00:51:49
• The key trick was training on tiny 64x64 images first, then fine-tuning briefly on full 224x224 images since the model already knew what a dog looked like.
  00:50:47
• Jeremy claims none of the big deep learning breakthroughs of the last 20 years required multiple GPUs, and all major breakthroughs of the next 20 years will be doable on a single GPU.
  00:54:59
• His team developed GAN-level image results without using GANs at all, and Jason Antic's DeOldify colorizes whole movies on a single home GPU.
  00:55:30
• Jeremy's only paper introduced successful transfer learning to NLP (ULMFiT) and the prototype took a couple of days yet smashed state-of-the-art.
  00:43:23
• Leslie Smith discovered super-convergence (training 10x faster with 10x higher learning rates) but couldn't publish because reviewers demanded an explanation.
  01:00:08
• Across teaching thousands of domain experts, the single differentiator between success and failure is tenacity, not talent.
  01:20:43

• The K programming language (from the APL family) is so compact it fits inside the L3 cache of a CPU, wins data-processing benchmarks, and costs about $100,000 per CPU, used by elite hedge funds.
  00:11:28
• Google deliberately made TPUs almost entirely unprogrammable because they feared people would learn their IP secrets.
  00:22:52
• There is roughly a 10x global shortage of doctors, and training enough the traditional way would take about 300 years.
  00:24:25
• Outside South Africa, the entire African continent has only five pediatric radiologists.
  00:25:33
• HIPAA stands for portability, not privacy, and was designed to enable thoughtful data sharing, but hospital lawyers interpret its gray areas conservatively.
  00:31:21
• Pete Worden (now at Google) once spent about a year figuring out how to train ImageNet in a converted granny flat behind his house.
  00:48:09
• Spaced repetition was discovered by psychologist Ebbinghaus roughly 150 years ago by memorizing random sequences of letters on cards.
  01:32:45
• Jeremy uses Anki only for Chinese, and after a three-year medical break the characters were still retained and far faster to relearn.
  01:36:23
• Jeremy built FastMail and wrote everything in Perl, including his own monitoring system and web framework because none existed in the late 90s.
  00:13:31