Your model works. It is just too big to ship. You have two standard paths to shrink it, and they are nothing alike. Quantization is compression. Distillation is re-education[reference:42]. One shrinks the same brain; the other builds a smaller one that imitates the original.
Quantization: Compression
Quantization takes your trained model and re-encodes its weights at lower precision. Instead of storing each weight as a 32-bit float (FP32), you store it as a 16-bit half (FP16), an 8-bit integer (INT8), or even a 4-bit integer (INT4).
The trade-off is that the accuracy loss compounds with each quantization step. INT8 is usually transparent. INT4 can degrade your output quality if your model has learned fine-grained weights. If your hardware supports low-precision computation (INT8 Tensor Cores on NVIDIA GPUs), you get direct compute speedup.
Quantization is the right choice when you have a model that works well, you need a fast win, and you have no time or data to retrain. It is zero-effort compression.
Distillation: Re-Education
Distillation is different. You take your big, trained model (the teacher) and train a new, smaller model (the student) to produce the same outputs. During training, the student learns not just the original task labels, but also the soft probabilities and internal representations the teacher produces. The teacher acts as a training signal, often richer and more informative than the raw labels alone.
Distillation requires the teacher model, training data, and time to train the student. Because you are training a new model, you have full control over its architecture, depth, and capacity. You can make it dramatically smaller than quantization alone would allow.
Distillation is the right choice when you need a permanently smaller model, you have the teacher and training data available, and you can afford the training time. The result is a genuinely lighter model, not just a compressed version of the original.
The Combined Approach
You can also combine both techniques. This gives you the best of both worlds: a model that is architecturally small (from distillation) compressed to low precision (from quantization). The student, being smaller to begin with, often handles quantization better than the original teacher would.
For a production LLM API, distill then quantize the result for the best latency and throughput profile. Distill first to pick the right architecture, then quantize to shrink it further.
Performance Considerations
- Time to First Token. How long until the model generates the first token. Driven by the prefill pass over the full prompt.
- Time Per Subsequent Token. How long each subsequent token takes. Driven by the decode pass on one token at a time.
Research shows that quantization provides the best overall trade-off between retained performance and efficiency, whereas distillation yields strong runtime acceleration gains at high computational cost[reference:43]. Quantization offers the greatest standalone compression[reference:44].
When to Use Each
Key Takeaway
Quantization is for when you need to ship now. Distillation is for when you need to ship smaller forever. And if you need the smallest, fastest inference possible, do both.