In February 2026, Andrej Karpathy released what he called an art project: a single file of roughly 200 lines of pure Python with zero dependencies that trains and runs a GPT. No PyTorch, no NumPy, no TensorFlow. Just Python's built-in math and random modules, plus a custom autograd engine built from scratch. The script trains on 32,000 names and learns to generate new ones like "kamon" and "karai" that sound plausibly human. The opening comment says it all: "This file is the complete algorithm. Everything else is just efficiency."
This is not a toy. The same mathematical operations inside these 200 lines are what power ChatGPT, Claude, Gemini, and every other transformer-based language model. The difference is scale and speed, not algorithm.
What the Script Actually Does
The pipeline is deceptively simple. First, it loads a dataset of documents. In this case, 32,000 names, one per line. A basic tokenizer maps each unique character to an integer ID. With 26 lowercase letters plus a special beginning-of-sequence token, the vocabulary is just 27 symbols. Then comes the model itself: a tiny transformer with 4,192 parameters, complete with token embeddings, position embeddings, multi-head self-attention, an MLP block, and residual connections. The training loop feeds tokens one at a time, computes cross-entropy loss, backpropagates gradients through the custom autograd engine, and updates weights with Adam. After about a minute on a laptop, the loss drops from roughly 3.3 (random guessing) to around 2.37, and the model starts producing coherent names.
The Autograd Engine: Where the Real Magic Lives
The most mathematically intense part is the scalar autograd engine. Each number in the computation is wrapped in a Value object that tracks how it was computed and knows its local derivative. When you add or multiply Value objects, the result is a new Value that remembers its parents and the gradient of the operation. The backward() method walks this graph in reverse topological order, applying the chain rule at each step. If the loss depends on a parameter through multiple paths, gradients accumulate. This is the exact same algorithm PyTorch runs, just on scalars instead of tensors. Algorithmically identical, significantly smaller, and of course far less efficient.
From Names to Conversations
The generated names do not look like much. But from ChatGPT's perspective, your entire conversation is just a longer, funnier-looking document. When you type a prompt, the model's response is statistical document completion, one token at a time. microGPT makes this literal. You can read every operation, trace every gradient, and watch 4,192 numbers learn the statistical regularities of English names.
What Production Models Add
The gap between microGPT and GPT-4 is enormous in engineering, not algorithm. Production models use subword tokenizers like BPE with vocabularies of ~100,000 tokens instead of 27 characters. They run on GPUs processing millions of scalars in parallel. They have hundreds of billions of parameters, 100+ layers, and embedding dimensions in the thousands. Modern additions like RoPE positional embeddings, grouped query attention, and mixture-of-experts layers improve efficiency but do not change the core structure: attention for communication, MLP for computation, interspersed on a residual stream.
Why This Matters
Karpathy's project is the culmination of a decade-long obsession to simplify LLMs to their bare essentials. It follows micrograd, makemore, and nanogpt, each stripping away another layer of abstraction. The result is a pedagogical tool that makes the invisible visible. You can modify the dataset, train longer, or grow the model size, and watch the effects directly. For anyone who has called model.fit() without understanding what happens inside, microGPT is the antidote. It proves that the "magic" of large language models is just math, scaled up.