In 1989, Yann LeCun and his colleagues published a paper that is now recognized as the earliest real-world application of a neural net trained end-to-end with backpropagation. The dataset was tiny: 7,291 grayscale images of digits at 16x16 pixels. The network was tiny: only about 1,000 neurons. But the paper reads remarkably modern. It lays out a dataset, describes the architecture, loss function, and optimization, and reports experimental error rates. Everything is recognizable as a modern deep learning paper, except it is from 33 years ago.
Reproducing the 1989 Paper
Andrej Karpathy set out to reproduce the paper, partly for fun and partly to use the exercise as a case study on the nature of progress in deep learning. He re-implemented everything in PyTorch. The original network was implemented in Lisp. The paper is in French, but from the syntax, it looks like you could specify neural nets using a higher-level API similar to what you would do in PyTorch today.
An exact reproduction is not possible because the original dataset has been lost to time. Karpathy had to simulate it using the larger MNIST dataset, scaling 28x28 digits down to 16x16 pixels with bilinear interpolation and drawing the correct number of training and test examples.
The Speedup. The original network trained for three days on a workstation. Karpathy's implementation on a MacBook Air M1 CPU crunched through it in about 90 seconds. That is a 3,000X naive speedup. Further gains are likely possible by switching to full-batch optimization and utilizing a GPU. The tiny network, with only 9,760 parameters, 64,000 MACs, and 1,000 activations, is actually too small to efficiently use an A100 GPU with per-example SGD.
Reproducing the Numbers. The original paper reported a test error of 5.00% with 102 misses. Karpathy's reproduction achieved 4.09% with 82 misses. Close, but not exact. The discrepancies likely come from the simulated dataset, ambiguous weight initialization details, formatting errors in the PDF, and a sparse connectivity structure that the paper brushed over.
Modern Improvements
Karpathy then asked: how much can we improve on the original result using modern understanding and 33 years of R&D?
Loss Function. The original modeled classification as MSE regression into targets of -1 or +1, with output neurons using tanh non-linearity. Karpathy swapped in a standard softmax cross-entropy loss. This change dramatically improved training error, completely overfitting the training set.
Optimizer. He switched from SGD to AdamW with a learning rate of 3e-4, decaying to 1e-4 over training. This gave a slightly improved result.
Data Augmentation. He introduced a simple strategy of shifting input images by up to one pixel horizontally or vertically. This simulates an increase in the dataset size, so he increased the number of passes from 23 to 60. The test error dropped to 2.19%.
Dropout and ReLU. He added weak dropout of 0.25 before the layer with the largest number of parameters and swapped all non-linearities from tanh to ReLU. The test error dropped further to 1.59%, with only 32 mistakes out of 2,007.
In summary, using modern techniques, Karpathy was able to cut the error rate by about 60% while keeping the dataset and inference latency unchanged. The gain did not come free: training time increased almost fourfold, which would have pushed the 1989 training time from three days to almost 12.
Scaling Up the Dataset
Another approach to improving performance would have been to scale up the dataset. Using the fact that all of MNIST is available, Karpathy scaled the training set by about 7X, from 7,291 to 50,000 examples. Leaving the baseline training running for 100 passes showed improvement from the added data alone. Combining this with modern innovations gave the best performance yet: a test error of just 1.25% with only 24 misses.
Lessons for 2022 and Beyond
What does this exercise tell us about deep learning in 2022? And what would a time traveler from 2055 think?
First, not much has changed on the macro level in 33 years. We are still setting up differentiable neural net architectures and optimizing them end-to-end with backpropagation and stochastic gradient descent. Everything reads familiar, except it is smaller.
The 1989 dataset is a baby by today's standards. Today's vision datasets contain a few hundred million high-resolution color images, roughly 100,000,000X more pixel data at the input. The neural net is also a baby. Modern vision models have a few billion parameters, about 1,000,000X more. Natural language models reach into trillions.
Karpathy projects that a time traveler from 2055 would see today's datasets and models as a joke, both around 10,000,000X larger. They would train today's state-of-the-art models in about a minute on a personal device as a weekend fun project. They would find that today's models are not optimally formulated, and that changing some details could about halve the error. They would note that today's datasets are too small, and that further gains are not possible without expanding computing infrastructure and investing in R&D.
The Rise of Foundation Models
But the most important trend is that training a neural network from scratch on a target task is quickly becoming outdated due to fine-tuning, especially with the emergence of foundation models like GPT. These models are trained by only a few institutions with substantial computing resources. Most applications are achieved via lightweight fine-tuning, prompt engineering, or distillation into smaller, special-purpose networks.
Karpathy expects this trend to intensify. In its most extreme extrapolation, you will not want to train any neural networks at all. In 2055, you will ask a 10,000,000X-sized neural net megabrain to perform some task by speaking to it in English. And if you ask nicely enough, it will oblige. Yes, you could train a neural net too. But why would you?