Knowledge Distillation: How Big Models Train Smaller Ones

Knowledge Distillation in AI is a powerful method where large models (teacher models) transfer their knowledge to smaller, efficient models (student models). This technique enables AI to retain high performance while reducing computational costs, speeding up inference, and facilitating deployment on resource-constrained devices like mobile phones and edge systems. By mimicking the outputs of teacher models, student models deliver lightweight, optimized solutions ideal for real-world applications. Let’s explore how knowledge distillation works and why it’s transforming modern AI.

1. What Is Knowledge Distillation?

Short answer: knowledge distillation is a model compression technique in which a small student model is trained to reproduce the outputs (and sometimes the internal representations) of a large, already-trained teacher model. The student ends up much smaller and faster while keeping most of the teacher’s accuracy.

Knowledge distillation is a technique where a large model (Teacher Model) transfers its knowledge to a smaller model (Student Model). The goal is to compress the large model’s capabilities into a lightweight version that is faster, more efficient, and easier to deploy, while retaining high performance.

Think of a teacher (large model) simplifying complex ideas for a student (small model). The teacher provides not just the answers but also insights into how the answers were derived, allowing the student to replicate the process efficiently.

The illustration from Knowledge Distillation: A Survey explained it:

Another figure is from A Survey on Knowledge Distillation of Large Language Models:

2. Why Is Knowledge Distillation Important?

Large models (e.g., GPT-4) are powerful but have significant limitations:

High Computational Costs: Require expensive hardware and energy to run.
Deployment Challenges: Difficult to use on mobile devices or edge systems.
Slow Inference: Unsuitable for real-time applications like voice assistants.

Knowledge distillation helps address these issues by:

Reducing Model Size: Smaller models require fewer resources.
Improving Speed: Faster inference makes them ideal for resource-constrained environments.
Maintaining Accuracy: By learning from large models, smaller models can achieve comparable performance.

3. How Does Knowledge Distillation Work?

The process involves several key steps:

(1) Train the Teacher Model

A large model is trained on a comprehensive dataset to achieve high accuracy and generalization.

(2) Generate Soft Targets

The teacher model produces outputs with detailed probability distributions.
For example, when classifying an image, instead of just saying “cat,” the teacher might output:
- Cat: 80%
- Dog: 15%
- Fox: 5%.
These soft targets provide rich information about how the teacher distinguishes between categories.

(3) Train the Student Model

The smaller model learns from both the teacher’s soft targets and the original data.
By mimicking the teacher’s outputs, the student absorbs the distilled knowledge without requiring as much capacity.

(4) Evaluate and Optimize

The student model’s performance is validated and fine-tuned to ensure it meets the desired accuracy and efficiency.

Soft Targets, Temperature and the Distillation Loss

The recipe most people mean by “knowledge distillation” comes from Hinton, Vinyals and Dean, Distilling the Knowledge in a Neural Network (2015). Two ideas make it work:

Temperature. A teacher’s softmax is usually very confident (99.9% “cat”), which hides the useful part: that “dog” is far more likely than “truck”. Dividing the logits $z_i$ by a temperature $T > 1$ before the softmax spreads the probability out:

p_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}

At $T = 1$ you get the normal softmax; at $T = 2$ – $5$ the “dark knowledge” about similar classes becomes visible.

A two-part loss. The student is trained on a weighted sum of the normal loss on the true labels and a loss that pulls its softened outputs toward the teacher’s:

\mathcal{L} = \alpha \cdot \mathrm{CE}(y, p_s) + (1 - \alpha) \cdot T^2 \cdot \mathrm{KL}\big(p_t^{(T)} \,\|\, p_s^{(T)}\big)

Here $p_t^{(T)}$ and $p_s^{(T)}$ are the teacher and student distributions at temperature $T$ , and $\alpha$ balances the two terms. The $T^2$ factor keeps the gradient of the soft term on the same scale when you change $T$ . At inference time the student runs at $T = 1$ like any other model.

Types of Knowledge Distillation

The survey Knowledge Distillation: A Survey groups methods by what knowledge is transferred and how the training is organised.

By the knowledge that is transferred:

Response-based: the student matches the teacher’s final outputs (the soft targets above). Simplest and most common.
Feature-based: the student also matches the teacher’s intermediate activations or attention maps, which helps when the student is much smaller.
Relation-based: the student matches relationships, such as how the teacher measures similarity between pairs of examples.

By the training scheme:

Offline distillation: a frozen, pre-trained teacher teaches the student. This is the classic setup.
Online distillation: teacher and student (or several peers) are trained at the same time and learn from each other.
Self-distillation: a model teaches itself, for example deeper layers teaching shallower ones, or a previous checkpoint teaching the next.

Knowledge Distillation in Large Language Models

Distillation is how many of the small models you can run on a laptop were made:

DistilBERT (Sanh et al., 2019) is distilled from BERT. The paper reports it is 40% smaller and 60% faster while keeping about 97% of BERT’s language-understanding performance.
DeepSeek-R1 distilled models (January 2025): DeepSeek fine-tuned smaller open models from the Qwen and Llama families on reasoning samples generated by DeepSeek-R1, releasing students from 1.5B to 70B parameters.
Black-box distillation: when you can only call a teacher through an API, you cannot see its logits. Instead you generate a dataset of teacher answers (often with step-by-step reasoning) and fine-tune the student on it. This is sequence-level distillation, and it is what most “distilled” LLMs today actually do. Check the teacher’s terms of service first: some providers forbid using outputs to train competing models.

The Survey on Knowledge Distillation of Large Language Models covers these LLM-specific methods in depth.

Knowledge Distillation vs Quantization vs Pruning

All three make a model cheaper to run, and they are often combined (distill first, then quantize the student).

Technique	What changes	Needs training?	Typical result
Knowledge distillation	A new, smaller architecture learns from the big model	Yes, the student is trained	Far fewer parameters; some accuracy loss on hard tasks
Quantization	The same weights are stored with fewer bits (e.g. FP16 → INT8 / 4-bit)	Usually no (post-training), or light	Smaller memory footprint, faster inference, same architecture
Pruning	Unimportant weights, heads or layers are removed	Usually a short fine-tune afterwards	Sparser or thinner model; speed-up depends on hardware support

4. A Simple Example: The Classroom Analogy

Without Distillation: A small model learns directly from raw data, like a student relying solely on a textbook without guidance.
With Distillation: The teacher (large model) explains not only the answers but also why certain conclusions are drawn. The student absorbs these nuanced insights, leading to better understanding.

Probabaly below figure from Knowledge Distillation : Simplified can help:

5. Real-World Applications of Knowledge Distillation

(1) Lightweight AI on Edge Devices

Small, distilled models are deployed on smartphones, IoT devices, and embedded systems.
Example: A distilled CLIP model for image classification on mobile.

(2) Real-Time Applications

Faster inference is crucial for speech recognition or recommendation systems in real-time scenarios.
Example: Voice assistants using distilled models for quick responses.

(3) Multitask Learning

Combine multiple teacher models into one small model capable of handling various tasks.
Example: A single model for both translation and sentiment analysis.

6. Challenges in Knowledge Distillation

(1) Knowledge Loss

Small models may fail to replicate the full depth of understanding from large models, especially for complex tasks.

(2) Computational Overhead

Generating soft targets from a teacher model can be resource-intensive when working with large datasets.

(3) Task-Specific Needs

Different tasks require different knowledge. Adapting distilled models to specific tasks remains a research challenge.

7. One-Line Summary

Knowledge distillation compresses the “wisdom” of large models into smaller, efficient ones, enabling faster, cost-effective AI without sacrificing accuracy.

FAQ

What is knowledge distillation in simple terms?

A big, accurate “teacher” model shows a small “student” model not just the right answers but how confident it is about every possible answer. The student learns from those richer signals and ends up nearly as good while being much smaller and faster.

What is the difference between the teacher and the student model?

The teacher is a large model that has already been trained and is only used to produce targets. The student is the smaller model you actually deploy; it is trained on the teacher’s outputs plus the original labels.

Why use soft targets instead of hard labels?

A hard label says only “cat”. A soft target says “80% cat, 15% dog, 5% fox”, which also tells the student which classes look alike. That extra information lets a small model learn more from each example.

What does temperature do in knowledge distillation?

Temperature divides the logits before the softmax. A higher temperature makes the probability distribution smoother so the student can see the teacher’s preferences among the wrong answers. Values between about 2 and 5 are common starting points; it is a hyperparameter you tune.

Is knowledge distillation the same as fine-tuning?

No. Fine-tuning adapts an existing model to new data. Distillation trains a (usually smaller) model to imitate another model. In practice LLM distillation is often done by fine-tuning a small model on text generated by a large one, which is why the terms get mixed up.

Is knowledge distillation better than quantization?

They solve different problems and work well together. Quantization keeps the architecture and stores weights in fewer bits, with no or little training. Distillation produces a genuinely smaller architecture but needs a training run. A common pipeline is to distill and then quantize the student.

Final Thoughts

Knowledge distillation bridges the gap between large, powerful models and real-world deployment. By making AI both smarter and leaner, this technique is transforming applications from edge devices to real-time systems. Next time you use a quick AI assistant on your phone, think about the distilled knowledge powering it. And when your student model is ready to serve behind an API, my Ray Serve guide covers model serving with autoscaling and request batching. Stay tuned for more insights tomorrow!

# Knowledge Distillation: How Big Models Train Smaller Ones