Andrej Karpathy set out to reproduce a paper he considers a landmark: Yann LeCun's 1989 study, widely regarded as the earliest real-world application of a neural network trained end-to-end with backpropagation. Despite using a dataset of just 7,291 16x16 grayscale digit images and a network of roughly 1,000 neurons, the paper's structure, covering architecture, loss function, optimization, and error reporting, reads as recognizably modern.
Rebuilding a 33-Year-Old Result
Karpathy reimplemented the network in PyTorch, since the original ran in a Lisp-based framework called Lush. Because the original 1989 dataset appears to be lost, he simulated it by downsampling MNIST digits to 16x16 pixels. His reproduction landed close to, but not exactly matching, the original numbers, with a 4.09% test error rate versus the paper's reported figures, a gap he attributes partly to ambiguous weight-initialization details in a possibly OCR-damaged PDF.
What 33 Years of Progress Actually Buys You
The more striking part of the exercise is what happened when Karpathy applied three decades of accumulated deep learning knowledge to the identical, unchanged dataset. Swapping the original MSE regression loss for a modern softmax classifier, replacing hand-tuned SGD with AdamW, adding simple data augmentation, and introducing dropout with ReLU activations cut the test error from roughly 4.09% to 1.59%, nearly a 60% reduction in mistakes. Separately, scaling the training set alone (using MNIST's larger pool) pushed error down further still.
The Real Lesson Isn't the Digits
Karpathy draws out three practical conclusions: hardware speedups alone delivered a roughly 3,000x training-time improvement (three days down to about 90 seconds), modern algorithmic knowledge alone could roughly halve the error rate on unchanged data, and further gains would have required a bigger network and more compute, not just cleverer ideas.
He extends the exercise into speculation about 2055, arguing that if the same pattern holds, future practitioners will look back at 2022-era models as comparably small, and that the entire practice of training networks from scratch is already giving way to fine-tuning and prompting large foundation models rather than building task-specific networks from zero. It's a framing that treats scaling laws and foundation-model reuse not as passing trends but as the continuation of a pattern already visible across three decades of neural network research.