Vivienne Sze: MIT professor specializing in energy-efficient, high-performance hardware and algorithms for machine learning, computer vision, and robotics. A world-class researcher working to close the energy-efficiency gap between AI systems and the human brain.
What this episode covers
In this MIT Deep Learning Series lecture hosted by Lex Fridman, Vivienne Sze gives a broad overview of efficient computing for deep learning, robotics, and AI. She explains that the explosive growth in compute demand and its carbon footprint, combined with the slowdown of Moore's Law and Dennard scaling, forces a turn toward specialized hardware. Her central theme is that power is dominated by data movement rather than computation, so the key to efficiency is reducing how often and how far data travels. She walks through hardware accelerators (the Eyeriss and Navion chips), algorithmic techniques like pruning and reduced precision, and joint hardware-algorithm co-design tools like NetAdapt. She closes with applications beyond neural nets, including robot localization, super-resolution, and low-cost in-home monitoring of neurodegenerative disease via eye movements on a phone.
The guest's own work
Efficient Processing of Deep Neural Networks (survey paper / forthcoming book)
Vivienne Sze and Joel Emer
“I want to point you first to this survey paper that we've developed this with my collaborator Joel Emmer Tom really kind of covers what is the different techniques”— Vivienne Sze
Efficient Processing of Deep Neural Networks (overview paper, ~30 pages, expanding into a book)
Vivienne Sze
“this is an overview paper that's about 30 pages and what we're currently expanding it into a book so if you're interested in this topic I would encourage you to visit these resources”— Vivienne Sze
Also referenced (named, not recommended)
iPhone 6
Apple
“measured on a subject on an iPhone 6 which is obviously under $1,000 way cheaper now compared to a phantom camera shown here in blue”— Vivienne Sze
iPhone
Apple
“we actually captured some footage you know on an iPhone and showed the you know real-time depth estimation on an iPhone itself and you can do achieved about 40 frames per second”— Vivienne Sze
NVIDIA Jetson TX2 GPU
NVIDIA
“the plot on the x axis here is the frame rate on a Jetson th to GPU this is a magic measure with the batch size of one with 32-bit float”— Vivienne Sze
Google TPU
“typical types of work that are popular that use this type of kind of data flow or weight stationary data flow are things like the TPU from Google”— Vivienne Sze
NVIDIA NVDLA accelerator
NVIDIA
“the envy de la accelerator from it video another approach that people take”— Vivienne Sze
ARM Cortex-A7 embedded processor
ARM
“algorithms that can run on VGA at 30 frames per second on our cortex ace which is a super low-cost embedded processor”— Vivienne Sze
Big reveals from this episode
- Compute required for deep learning grew exponentially, by over 300,000 times, to drive accuracy gains in recent years.
- The key contrarian insight: power is dominated by data movement, not the computations themselves.
- Reading data from off-chip DRAM costs two orders of magnitude more energy than performing the actual multiply-and-accumulate.
- AlexNet and VGG cost two to three orders of magnitude more energy than HOG features for object detection.
- Energy-aware pruning (removing high-energy weights first) yields a 3.7x energy reduction versus 2x for traditional magnitude-based pruning at the same accuracy.
- Their FastDepth approach runs real-time monocular depth estimation at about 40 frames per second on an iPhone.
- Eye-reaction-time measurements on a sub-$1,000 iPhone 6 matched the distributions of an expensive phantom camera, enabling low-cost in-home disease monitoring.
Worth remembering
- Training a neural net can have a carbon footprint orders of magnitude greater than flying across North America or an average human life.
- A self-driving car can consume over 2000 watts just for computation, generating heat that often needs water cooling.
- Handheld devices are typically limited to under a watt of power due to heat dissipation constraints.
- The multiply-and-accumulate operation accounts for over 90% of the computation in neural networks.
- Every multiply-and-accumulate requires four memory accesses, a four-to-one ratio of memory access to compute.
- Skipping multiplications where an input is zero can reduce power consumption by almost 50%.
- HOG features are actually more energy-efficient than video compression for understanding a pixel.
- Their hardware can compute mutual information over a 200m x 200m map in under a second, a 100x speedup over a CPU.
- Their depth-sensing method turns the power-hungry depth sensor on only about 11% of the time with only ~0.7% mean relative error.