In 1989, Yann LeCun and his colleagues published a paper of historical significance. It is considered the earliest real-world application of a neural network trained end-to-end with backpropagation. The dataset was tiny by today's standards: just 7,291 16x16 grayscale images of handwritten digits. The network was equally small, with only about 1,000 neurons and 9,760 parameters. Yet, if you read that 1989 paper today, it feels remarkably modern. It lays out a dataset, describes the neural net architecture, loss function, optimization, and reports experimental classification error rates.
I set out to reproduce this paper. First, because it's fun. Second, to use the exercise as a case study on the nature of progress in deep learning.
Reproducing the 1989 State of the Art
The original network trained for 3 days on a workstation. I ran my re-implementation in PyTorch on a modern MacBook Air (M1) CPU, and it crunched through the 23 passes over the training set in about 90 seconds. That's roughly a 3,000x naive speedup. The software design is also far more sophisticated: we now have fast C/CUDA tensor libraries, an autograd engine that tracks the compute graph, and a high-level Python API for common operations.
However, an exact reproduction is impossible. The original dataset is lost to time, so I simulated it using the larger MNIST dataset. There were also ambiguities in the paper regarding weight initialization and a sparse connectivity structure. My reproduced baseline achieved a test error of 4.09%, while the original claimed around 5%, so I was in the ballpark.
The Power of Modern Knowledge
This is where things get interesting. In 33 years, the deep learning field has accumulated a wealth of new techniques. How much could we improve the 1989 result without scaling up the model or dataset?
The first change was swapping the MSE loss with tanh output for a standard softmax function. This eliminated a significant optimization difficulty and allowed the model to overfit the training set completely.
Next, I switched from SGD to AdamW. The modern optimizer, with its learning rate of 3e-4 and weight decay, is almost always a strong baseline and needs little tuning. This gave a slight improvement.
Then came the bigger gains. I introduced data augmentation: shifting images by up to 1 pixel. This is a standard modern technique to fight overfitting. I also added dropout of 0.25 before the largest layer and swapped tanh for ReLU to work better with dropout. These combined modern techniques cut the test error from ~4% down to just 1.59%. That's a 60% reduction in errors, dropping the mistake count from about 80 to 32.
The gains didn't come free. Training time increased from 3 days to almost 12 if we were still in 1989. But inference latency was unchanged, and the model was significantly more accurate.
Scaling Up to Modern Standards
Further improvement is still possible. Simply scaling up the training dataset from 7,291 examples to the full 50,000 MNIST training examples also helped. Combining data scaling with modern innovations yielded a final test error of just 1.25% .
The lesson is clear. If I time-traveled to 1989, I would be able to cut the error rate by 60% just by applying modern techniques. My upper bound would not be the model architecture, but the compute available. To go further, I would need a bigger computer.
What Does This Mean for 2022?
Suppose the lessons of this exercise remain invariant over time. What would a time traveler from 2055 think about our 2022 networks?
- No fundamental changes: 2055 neural nets are the same as 2022 nets on the macro level, just bigger.
- Our models are jokes: Today's models and datasets are about 10,000,000X smaller than what's to come.
- Training speed: Training a 2022 state-of-the-art model will be a weekend fun project on a personal device in 2055.
- Optimization: Today's models are not optimally formulated. Small changes in loss function, augmentation, or optimizer can halve the error.
- The real barrier: The real gains come from scaling compute and investing in R&D, which is harder than just applying existing techniques.
The Biggest Trend: The Rise of the Megabrain
The most important trend is that the entire setting of training a neural network from scratch on a target task is becoming outdated. The emergence of foundation models like GPT means that the heavy lifting is done by a few institutions. Most applications are now just lightweight fine-tuning, prompt engineering, or distillation.
In its most extreme extrapolation, you won't want to train any neural networks at all. In 2055, you'll ask a 10,000,000X-sized neural net megabrain to perform a task by speaking to it in English. And if you ask nicely enough, it will oblige.