Andrej Karpathy has released microgpt, a single Python file of roughly 200 lines that trains and runs a GPT-style language model with zero external dependencies. The project is billed as an art piece as much as a teaching tool, the culmination of Karpathy's earlier micrograd, makemore, and nanoGPT projects compressed into their bare algorithmic essence.
What's Actually Inside the File
The script contains every component needed to go from raw text to a working model: a dataset of documents (32,000 names, one per line), a character-level tokenizer, a from-scratch autograd engine, a GPT-2-style architecture with RMSNorm and ReLU instead of LayerNorm and GeLU, an Adam optimizer, and both training and sampling loops. Karpathy says he 'cannot simplify this any further.'
The tokenizer is intentionally primitive, assigning one integer per unique character rather than using subword merging like production tokenizers such as GPT-4's. A special beginning-of-sequence token wraps each name during training, giving the model 27 total vocabulary entries. The autograd engine, built around a scalar-valued class, mirrors PyTorch's autograd.Function mechanics but operates one number at a time instead of on batched tensors.
Training on a Laptop, Not a Cluster
With just 4,192 parameters, the model trains in about a minute on a MacBook, watching loss fall from roughly 3.3 to 2.37 over 1,000 steps. Once trained, it generates new, plausible-sounding names it has never seen, the same underlying mechanism, Karpathy notes, that produces a ChatGPT response: statistical completion of a "document" that happens to be a conversation.
Why the Toy Scale Matters
What separates microgpt from a typical tutorial is its explicit mapping between the toy model and production systems. Karpathy walks through each simplification: character tokens versus byte-pair encoding, scalar autograd versus tensor operations on GPUs, thousands of parameters versus hundreds of billions, and a single training document at a time versus massive batched runs across GPU clusters.
The project lands as open-source AI tooling has moved toward assembling ever-larger systems from prebuilt libraries, making a fully transparent, from-scratch implementation a notably different kind of resource for people trying to understand what actually happens inside a language model rather than just how to call one through an API.
For developers looking to build genuine intuition about transformers rather than just use them, spending an afternoon reading through microgpt line by line may do more than another framework tutorial ever could.