Deep Learning for Computer Vision (Andrej Karpathy, OpenAI)

This is a technical lecture by Andrej Karpathy covering deep learning for computer vision, focused on convolutional neural networks (CNNs). He traces the field's history from Hubel and Wiesel's 1960s cat experiments through Fukushima's neurocognitron, Yann LeCun's 1990s LeNet, and the 2012 AlexNet breakthrough that transformed the field. He explains the mechanics of convolutional, pooling, and fully connected layers, then walks through the evolution of winning ImageNet architectures (AlexNet, ZFNet, VGGNet, GoogLeNet, ResNet). The talk closes with practical guidance on hardware, software frameworks, architecture selection, hyperparameters, and distributed training, followed by an extended audience Q&A.

• By 2016 ImageNet top-5 error dropped to about 3.57%, roughly matching or beating estimated human accuracy of 2-5%.
  00:06:42
• CNNs replaced multi-page hand-engineered feature extraction pipelines with end-to-end trained networks, drastically cutting code complexity.
  00:08:48
• Features learned by pre-training on ImageNet transfer remarkably well to entirely different datasets and tasks.
  00:09:51
• In ~20 years the two main algorithmic advances over LeNet were dropout and ReLU, both essentially one-line changes that set values to zero.
  00:36:15
• Residual networks (ResNet) won ImageNet 2015 and many other challenges, enabling far deeper networks via skip connections acting as a gradient superhighway.
  00:43:38
• ResNets can be made much shallower and wider and still work as well or better, suggesting depth alone is not the key.
  00:49:53
• Karpathy's practical advice: 'don't be a hero' - use pre-trained ImageNet models and fine-tune rather than designing custom architectures.
  01:03:26

• Spectrograms are 2D arrays, images 3D, videos 4D, and text can be treated as a 1D array of numbers.
  00:01:01
• Karpathy built a web interface to measure his own ImageNet accuracy by competing against a CNN, losing points mostly on dog-breed identification.
  00:07:13
• Some ImageNet test images are actually mislabeled, and ImageNet contains 50 different types of terriers.
  00:07:46
• CNN neurons learned to detect faces, wrinkles, and printed text on their own, despite labels only being provided at the final layer.
  00:30:33
• GoogLeNet has only 5 million parameters versus VGGNet's 140 million, mainly by discarding fully connected layers.
  00:42:00
• A 56-layer plain network performs worse than a 20-layer one even on training data, an optimization problem ResNet's skip connections solve.
  00:44:09
• NVIDIA's DGX-1 supercomputer cost about $130,000 at the time of the talk.
  01:00:48
• State-of-the-art networks were typically trained for a few weeks across four or eight GPUs costing about $1,000 each.
  01:05:27