In 2022, Andrej Karpathy set out to reproduce a paper of historical significance: Yann LeCun et al.'s 1989 "Backpropagation Applied to Handwritten Zip Code Recognition"[reference:20][reference:21]. To his knowledge, it's the earliest real-world application of a neural net trained end-to-end with backpropagation[reference:22]. Except for the tiny dataset (7,291 16x16 grayscale images of digits) and the tiny neural network (only 1,000 neurons), the paper reads remarkably modern today. It lays out a dataset, describes the architecture, loss function, and optimization, and reports classification error rates. It's all very recognizable—except it's from 33 years ago.
The Reproduction
Karpathy re-implemented everything in PyTorch. The original network was implemented in Lisp using the Bottou and LeCun 1988 simulator SN (later named Lush)[reference:23]. The original trained for 3 days on a workstation. Karpathy's implementation on a MacBook Air (M1) CPU crunched through it in about 90 seconds—a ~3,000x speedup. He also tried running it on an A100 GPU, but training was actually slower because the network is so tiny and SGD uses a single example at a time.
His reproduction got close to the original numbers (4.09% test error vs. 5.00%), but not exactly. The original dataset has been lost to time, so he had to simulate it using MNIST, scaling 28x28 digits down to 16x16. Other factors, like ambiguous weight initialization schemes and sparse connectivity structures that were "brushed over" in the paper, also contributed to the discrepancy.
Modern Improvements
Karpathy then asked: how much could we improve on the original result using 33 years of R&D? He tried several modern innovations:
- Swapping MSE for Softmax: The original used MSE regression into targets of -1 or +1 with tanh output neurons. Switching to softmax cross-entropy dramatically improved training, completely overfitting the training set.
- AdamW: Switching from SGD to AdamW with learning rate 3e-4 gave a slight improvement.
- Data Augmentation: Shifting input images by up to 1 pixel horizontally or vertically helped fight overfitting, dropping test error to 2.19%.
- Dropout and ReLU: Adding weak dropout of 0.25 and swapping tanh for ReLU brought test error down to 1.59%.
In total, he cut the error rate by about 60%, from ~80 to ~30 mistakes on the test set. The gains didn't come free—training time increased by about 4x—but inference latency was unaffected.
Scaling Up the Dataset
Using the fact that all of MNIST is available, Karpathy also tried scaling up the training set by ~7x (from 7,291 to 50,000 examples). Combining this with the modern innovations gave the best performance yet: just 24 mistakes on the test set, an error rate of 1.25%.
Lessons for the Future
Karpathy used the exercise as a case study on the nature of progress in deep learning. His takeaways:
- Not much has changed on the macro level: We're still setting up differentiable neural net architectures and optimizing them with backpropagation and SGD.
- Datasets and models are vastly larger: Today's vision datasets have ~100,000,000x more pixel data at the input, and models have millions of times more parameters.
- Speedups are dramatic: A 3-day training run in 1989 now takes 90 seconds on a laptop.
- Modern innovations can halve the error: With the same dataset and latency, modern tricks cut errors by 60%.
- Further gains require scale: Without bigger computers, progress stalls.
Extrapolating to 2055, Karpathy imagines a world where today's models look like a joke, where one can train a 2022 state-of-the-art model in ~1 minute on a personal device, and where the dominant paradigm is not training from scratch at all, but asking a 10,000,000x-sized neural net megabrain to perform tasks in plain English.