There's an enormous gap between the hype around billion-parameter models and the day-to-day experience of actually building a machine learning model. For someone like me, who's spent a decade obsessed with simplifying neural networks to their bare essentials, the sheer complexity of modern AI can feel like a wall.
That's the problem microgpt was built to solve. It is a single Python file of just 200 lines, with no external dependencies. It contains everything you need to understand the full algorithmic content of a GPT: a tokenizer, an autograd engine, a neural network architecture, an optimizer, and training and inference loops. Everything else in production AI is just efficiency. I think this script is beautiful, and it breaks perfectly across three columns in a text editor.
What's Inside the Black Box?
The fuel of a language model is just text. microgpt uses a dataset of 32,000 names, one per line. The goal is to learn the statistical patterns in this data and generate new, plausible-sounding names. The model will learn that names often start with consonants, that 'qu' tends to appear together, and that you rarely see three consonants in a row.
To the model, a "document" is just a sequence of tokens. From the perspective of an LLM, your conversation with ChatGPT is also just a funny looking "document". The model's response is just a statistical completion of that document.
The Autograd Engine: Learning by Nudging
The core of learning is understanding how to change a parameter to get a better answer. microgpt implements this from scratch with a class called Value. Think of it like a little Lego block. Every time you do math (like adding or multiplying), the Value remembers its inputs and the local derivative of that operation. The backward method then walks this graph backward, applying the chain rule of calculus to figure out how much each parameter contributed to the final loss.
The math is simple. If a car travels twice as fast as a bicycle, and the bicycle is four times as fast as a walking man, then the car travels eight times as fast as the man. The chain rule is just multiplying rates of change along a path. microgpt lets you see this in action, which is incredibly satisfying.
The Architecture: The GPT-2 Brain
The model architecture is a stripped-down version of GPT-2. It has a tiny 4,192 parameters and processes one token at a time. Here's how the forward pass works, step-by-step:
-
Token and Position Embeddings: The raw token ID (e.g., '5' for 'e') and its position in the sequence are looked up in learned tables and added together. This gives the model a representation of both the token and its place in the sequence.
-
Attention: This is where the model "looks" at the past. The current token projects a Query, while previous tokens have Keys and Values. The Query and Keys are compared (dot product) to see which past tokens are most relevant for predicting the next token. The weighted Values from those relevant past tokens are then aggregated to build the new representation.
-
MLP (Multilayer Perceptron): This is the "thinking" part. It's a simple two-layer network that projects the representation up to a larger size, applies a ReLU activation, and projects back down. It processes each position independently, unlike Attention which is the only communication mechanism between tokens.
-
Output: The final hidden state is projected to the vocabulary size of 27 to produce logits (scores). The higher the logit for a token, the more the model thinks it should come next.
Training and Sampling
The training loop is equally straightforward. It picks a document, wraps it with a special [BOS] token, and feeds the tokens one by one. At each position, the model outputs logits. We convert these to probabilities via softmax. The loss for that position is the negative log probability of the correct next token. The backward() call sends gradients all the way through the network, and an optimizer (Adam) nudges every parameter to lower the loss.
To sample a new name, we start with the [BOS] token. The model produces probabilities over the next token, and we sample according to that distribution. We feed that token back in and repeat until the model produces another [BOS] token (meaning it's done). A temperature parameter controls the randomness. Lower temperatures make the model more conservative, always picking its top choices. Higher temperatures produce more creative but potentially less coherent results.
From microgpt to ChatGPT
microgpt contains the complete algorithmic essence. Between this 200-line script and a production LLM like ChatGPT is just a long list of things that change, but none of them alter the core algorithm.
- Data: Instead of 32K names, it's trillions of tokens from the web.
- Tokenizer: Instead of single characters, it's subword tokenizers like BPE with ~100K tokens.
- Compute: Instead of pure Python, it's GPUs/TPUs running PyTorch tensors.
- Parameters: Instead of 4,192, it's hundreds of billions.
- Post-Training: The base model is fine-tuned with supervised data and reinforcement learning to become a chatbot.
But if you understand microgpt, you understand the algorithmic essence. You understand how it learns, how it makes predictions, and how sampling from a probability distribution is the same mechanism whether you're completing a name or writing an essay. The model doesn't understand truth; it only knows what sequences are statistically plausible. And that's the magic—and the limitation—of the technology.