GeekCoding101 make your way to geek

Knowledge Distillation: How Big Models Train Smaller Ones

What knowledge distillation is, how soft targets, temperature and the distillation loss work, the main types, LLM examples like DistilBERT and DeepSeek-R1 distills, and how it compares with quantization and pruning.

14 min read
Daily Ai Insight Issue 14 20241216
On this page1. What Is Knowledge Distillation?

Knowledge Distillation in AI is a powerful method where large models (teacher models) transfer their knowledge to smaller, efficient models (student models). This technique enables AI to retain high performance while reducing computational costs, speeding up inference, and facilitating deployment on resource-constrained devices like mobile phones and edge systems. By mimicking the outputs of teacher models, student models deliver lightweight, optimized solutions ideal for real-world applications. Let’s explore how knowledge distillation works and why it’s transforming modern AI.

1. What Is Knowledge Distillation?

Short answer: knowledge distillation is a model compression technique in which a small student model is trained to reproduce the outputs (and sometimes the internal representations) of a large, already-trained teacher model. The student ends up much smaller and faster while keeping most of the teacher’s accuracy.

Knowledge distillation is a technique where a large model (Teacher Model) transfers its knowledge to a smaller model (Student Model). The goal is to compress the large model’s capabilities into a lightweight version that is faster, more efficient, and easier to deploy, while retaining high performance.

Think of a teacher (large model) simplifying complex ideas for a student (small model). The teacher provides not just the answers but also insights into how the answers were derived, allowing the student to replicate the process efficiently.

The illustration from  Knowledge Distillation: A Survey explained it:

knowledge-distillation-01

Another figure is from  A Survey on Knowledge Distillation of Large Language Models:


2. Why Is Knowledge Distillation Important?

Large models (e.g., GPT-4) are powerful but have significant limitations:

  • High Computational Costs: Require expensive hardware and energy to run.
  • Deployment Challenges: Difficult to use on mobile devices or edge systems.
  • Slow Inference: Unsuitable for real-time applications like voice assistants.

Knowledge distillation helps address these issues by:

  • Reducing Model Size: Smaller models require fewer resources.
  • Improving Speed: Faster inference makes them ideal for resource-constrained environments.
  • Maintaining Accuracy: By learning from large models, smaller models can achieve comparable performance.

3. How Does Knowledge Distillation Work?

The process involves several key steps:

(1) Train the Teacher Model

A large model is trained on a comprehensive dataset to achieve high accuracy and generalization.

(2) Generate Soft Targets

  • The teacher model produces outputs with detailed probability distributions.
  • For example, when classifying an image, instead of just saying “cat,” the teacher might output:
    • Cat: 80%
    • Dog: 15%
    • Fox: 5%.
  • These soft targets provide rich information about how the teacher distinguishes between categories.

(3) Train the Student Model

  • The smaller model learns from both the teacher’s soft targets and the original data.
  • By mimicking the teacher’s outputs, the student absorbs the distilled knowledge without requiring as much capacity.

(4) Evaluate and Optimize

The student model’s performance is validated and fine-tuned to ensure it meets the desired accuracy and efficiency.

Soft Targets, Temperature and the Distillation Loss

The recipe most people mean by “knowledge distillation” comes from Hinton, Vinyals and Dean, Distilling the Knowledge in a Neural Network (2015). Two ideas make it work:

Temperature. A teacher’s softmax is usually very confident (99.9% “cat”), which hides the useful part: that “dog” is far more likely than “truck”. Dividing the logits ziz_i by a temperature T>1T > 1 before the softmax spreads the probability out:

pi=exp⁡(zi/T)∑jexp⁡(zj/T)p_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}

At T=1T = 1 you get the normal softmax; at T=2T = 2–55 the “dark knowledge” about similar classes becomes visible.

A two-part loss. The student is trained on a weighted sum of the normal loss on the true labels and a loss that pulls its softened outputs toward the teacher’s:

L=α⋅CE(y,ps)+(1−α)⋅T2⋅KL(pt(T) ∥ ps(T))\mathcal{L} = \alpha \cdot \mathrm{CE}(y, p_s) + (1 - \alpha) \cdot T^2 \cdot \mathrm{KL}\big(p_t^{(T)} \,\|\, p_s^{(T)}\big)

Here pt(T)p_t^{(T)} and ps(T)p_s^{(T)} are the teacher and student distributions at temperature TT, and α\alpha balances the two terms. The T2T^2 factor keeps the gradient of the soft term on the same scale when you change TT. At inference time the student runs at T=1T = 1 like any other model.


Types of Knowledge Distillation

The survey Knowledge Distillation: A Survey groups methods by what knowledge is transferred and how the training is organised.

By the knowledge that is transferred:

  • Response-based: the student matches the teacher’s final outputs (the soft targets above). Simplest and most common.
  • Feature-based: the student also matches the teacher’s intermediate activations or attention maps, which helps when the student is much smaller.
  • Relation-based: the student matches relationships, such as how the teacher measures similarity between pairs of examples.

By the training scheme:

  • Offline distillation: a frozen, pre-trained teacher teaches the student. This is the classic setup.
  • Online distillation: teacher and student (or several peers) are trained at the same time and learn from each other.
  • Self-distillation: a model teaches itself, for example deeper layers teaching shallower ones, or a previous checkpoint teaching the next.

Knowledge Distillation in Large Language Models

Distillation is how many of the small models you can run on a laptop were made:

  • DistilBERT (Sanh et al., 2019) is distilled from BERT. The paper reports it is 40% smaller and 60% faster while keeping about 97% of BERT’s language-understanding performance.
  • DeepSeek-R1 distilled models (January 2025): DeepSeek fine-tuned smaller open models from the Qwen and Llama families on reasoning samples generated by DeepSeek-R1, releasing students from 1.5B to 70B parameters.
  • Black-box distillation: when you can only call a teacher through an API, you cannot see its logits. Instead you generate a dataset of teacher answers (often with step-by-step reasoning) and fine-tune the student on it. This is sequence-level distillation, and it is what most “distilled” LLMs today actually do. Check the teacher’s terms of service first: some providers forbid using outputs to train competing models.

The Survey on Knowledge Distillation of Large Language Models covers these LLM-specific methods in depth.


Knowledge Distillation vs Quantization vs Pruning

All three make a model cheaper to run, and they are often combined (distill first, then quantize the student).

TechniqueWhat changesNeeds training?Typical result
Knowledge distillationA new, smaller architecture learns from the big modelYes, the student is trainedFar fewer parameters; some accuracy loss on hard tasks
QuantizationThe same weights are stored with fewer bits (e.g. FP16 → INT8 / 4-bit)Usually no (post-training), or lightSmaller memory footprint, faster inference, same architecture
PruningUnimportant weights, heads or layers are removedUsually a short fine-tune afterwardsSparser or thinner model; speed-up depends on hardware support

Try It: Knowledge Distillation in PyTorch (Runnable Example)

Here is the whole recipe from above in about 80 lines of PyTorch. It runs on a laptop CPU in a few seconds, so you can change the knobs and watch what happens.

The setup uses scikit-learn’s built-in digits dataset (1,797 handwritten 8×8 images, 10 classes) so nothing needs downloading:

  • Teacher: an MLP with 512 hidden units (38,410 parameters).
  • Student: the same shape with only 16 hidden units (1,210 parameters), then an even smaller one with 8 (610 parameters).
  • Each student size is trained twice from the same initial weights: once on the hard labels only, once with the distillation loss. The only difference between the two runs is the loss function.
  • Only 300 images are used for training (the rest is the test set), because distillation helps most when labeled data is scarce.
  • Everything is repeated on 5 different random splits, because a single run on a test set this size can swing by a point or more just from the split.

Prerequisites

  • Python 3.10 or newer (I ran it on Python 3.12.12, macOS, CPU only)
  • PyTorch and scikit-learn (I used torch 2.15.0 and scikit-learn 1.9.1)
Terminal window
python3 -m venv .venv
source .venv/bin/activate
pip install torch scikit-learn

The code

Save this as distill_digits.py:

"""Knowledge distillation on scikit-learn's digits dataset (8x8 images, 10 classes).
For each of 5 random splits: train a big teacher MLP, then train the same small
student twice from identical initial weights:
1) on hard labels only
2) with distillation (teacher's soft targets at temperature T, plus hard labels)
Runs on CPU in a few seconds.
"""
import statistics
import torch
import torch.nn as nn
import torch.nn.functional as F
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
T, ALPHA, TRAIN_SIZE, EPOCHS, SEEDS = 4.0, 0.7, 300, 300, range(5)
X_all, y_all = load_digits(return_X_y=True)
X_all = torch.tensor(X_all / 16.0, dtype=torch.float32)
y_all = torch.tensor(y_all)
def mlp(hidden):
return nn.Sequential(nn.Linear(64, hidden), nn.ReLU(), nn.Linear(hidden, 10))
def params(m):
return sum(p.numel() for p in m.parameters())
def distillation_loss(student_logits, teacher_logits, labels, T=T, alpha=ALPHA):
# Soft part: match the teacher's softened distribution. Multiply by T*T so
# gradients keep the same scale when T changes (Hinton et al., 2015).
soft = F.kl_div(
F.log_softmax(student_logits / T, dim=1),
F.softmax(teacher_logits / T, dim=1),
reduction="batchmean",
) * (T * T)
hard = F.cross_entropy(student_logits, labels)
return alpha * soft + (1 - alpha) * hard
def train(model, loss_fn, X, epochs=EPOCHS):
opt = torch.optim.Adam(model.parameters(), lr=1e-2)
for _ in range(epochs):
opt.zero_grad()
loss_fn(model(X)).backward()
opt.step()
return model
def run(seed, hidden):
# A small training set makes the gap between the two students visible
X_tr, X_te, y_tr, y_te = train_test_split(
X_all, y_all, train_size=TRAIN_SIZE, stratify=y_all, random_state=seed)
def acc(m):
with torch.no_grad():
return (m(X_te).argmax(1) == y_te).float().mean().item()
torch.manual_seed(seed)
teacher = train(mlp(512), lambda out: F.cross_entropy(out, y_tr), X_tr)
with torch.no_grad():
teacher_logits = teacher(X_tr)
torch.manual_seed(seed + 100)
baseline = train(mlp(hidden), lambda out: F.cross_entropy(out, y_tr), X_tr)
torch.manual_seed(seed + 100) # same initial weights as the baseline student
student = train(mlp(hidden),
lambda out: distillation_loss(out, teacher_logits, y_tr), X_tr)
return acc(teacher), acc(baseline), acc(student)
print(f"torch {torch.__version__} | train={TRAIN_SIZE} T={T} alpha={ALPHA} epochs={EPOCHS}")
print(f"teacher: 512 hidden, {params(mlp(512)):,} params")
for hidden in (16, 8):
rows = [run(s, hidden) for s in SEEDS]
print(f"\nstudent: {hidden} hidden, {params(mlp(hidden)):,} params")
print("seed teacher hard-only distilled gain")
for s, (t, b, d) in zip(SEEDS, rows):
print(f"{s:>4} {t:7.3f} {b:9.3f} {d:9.3f} {d - b:+.3f}")
mean = [statistics.mean(r[i] for r in rows) for i in range(3)]
wins = sum(r[2] > r[1] for r in rows)
print(f"mean {mean[0]:7.3f} {mean[1]:9.3f} {mean[2]:9.3f} {mean[2] - mean[1]:+.3f}"
f" (distilled better in {wins}/{len(rows)} seeds)")

Run it:

Terminal window
python distill_digits.py

What I got

This is the real output from my terminal. The script seeds every run, so on the same versions you should see the same numbers (other PyTorch versions or hardware can shift the last digit):

Terminal output of distill_digits.py: with 16 hidden units the distilled student averages 0.937 vs 0.933 for hard labels only, and with 8 hidden units 0.915 vs 0.905, better in 4 of 5 seeds each time

How to read it:

  • The gain is real but small on this toy problem. With 16 hidden units, distillation lifts mean test accuracy from 0.933 to 0.937 (+0.4 points). With 8 hidden units the gap grows to +1.0 point (0.905 → 0.915). In both cases the distilled student wins on 4 of the 5 splits, and it ties or loses on the others (seed 4 of the 8-unit student is 0.6 points worse). Do not expect the dramatic jumps you see in papers from an 8×8 digits demo.
  • Why it is small here: the teacher itself only reaches 0.944, so there is not much “dark knowledge” above what the labels already say, and a 16-unit MLP already fits digits well. Distillation pays off more when the student is much weaker than the teacher (the 8-unit row shows that trend), when labels are scarce or noisy, and on harder tasks with many similar classes.
  • Soft targets are F.softmax(teacher_logits / T). The teacher’s logits are computed once on the training set and reused every epoch, since the teacher is frozen.
  • Temperature T = 4 softens both distributions before they are compared with KL divergence. At T = 1 the teacher’s output is close to one-hot and the soft term adds little beyond the hard labels.
  • The T * T factor is the T2T^2 in the formula above. Without it, raising T would shrink the soft term’s gradients by roughly 1/T21/T^2 and silently change the balance between the two losses.
  • ALPHA = 0.7 puts 70% of the weight on the soft (teacher) term and 30% on the hard-label cross-entropy. Note the convention: in the code alpha weights the soft term, so it corresponds to 1−α1 - \alpha in the formula above. Both conventions are common; check which one a library uses before copying a value.

Things to try: set ALPHA = 0.0 (the “distilled” student becomes identical to the hard-label one, a useful sanity check), try T = 1 or T = 8, or change TRAIN_SIZE to 100 or 1000. In my own quick sweep over these knobs the average gain stayed in the 0 to +1 point range, which is an honest picture of what distillation does on a dataset this easy.


4. A Simple Example: The Classroom Analogy

  • Without Distillation: A small model learns directly from raw data, like a student relying solely on a textbook without guidance.
  • With Distillation: The teacher (large model) explains not only the answers but also why certain conclusions are drawn. The student absorbs these nuanced insights, leading to better understanding.

Probabaly below figure from Knowledge Distillation : Simplified can help:


5. Real-World Applications of Knowledge Distillation

(1) Lightweight AI on Edge Devices

  • Small, distilled models are deployed on smartphones, IoT devices, and embedded systems.
  • Example: A distilled CLIP model for image classification on mobile.

(2) Real-Time Applications

  • Faster inference is crucial for speech recognition or recommendation systems in real-time scenarios.
  • Example: Voice assistants using distilled models for quick responses.

(3) Multitask Learning

  • Combine multiple teacher models into one small model capable of handling various tasks.
  • Example: A single model for both translation and sentiment analysis.

6. Challenges in Knowledge Distillation

(1) Knowledge Loss

Small models may fail to replicate the full depth of understanding from large models, especially for complex tasks.

(2) Computational Overhead

Generating soft targets from a teacher model can be resource-intensive when working with large datasets.

(3) Task-Specific Needs

Different tasks require different knowledge. Adapting distilled models to specific tasks remains a research challenge.


7. One-Line Summary

Knowledge distillation compresses the “wisdom” of large models into smaller, efficient ones, enabling faster, cost-effective AI without sacrificing accuracy.


FAQ

What is knowledge distillation in simple terms?

A big, accurate “teacher” model shows a small “student” model not just the right answers but how confident it is about every possible answer. The student learns from those richer signals and ends up nearly as good while being much smaller and faster.

What is the difference between the teacher and the student model?

The teacher is a large model that has already been trained and is only used to produce targets. The student is the smaller model you actually deploy; it is trained on the teacher’s outputs plus the original labels.

Why use soft targets instead of hard labels?

A hard label says only “cat”. A soft target says “80% cat, 15% dog, 5% fox”, which also tells the student which classes look alike. That extra information lets a small model learn more from each example.

What does temperature do in knowledge distillation?

Temperature divides the logits before the softmax. A higher temperature makes the probability distribution smoother so the student can see the teacher’s preferences among the wrong answers. Values between about 2 and 5 are common starting points; it is a hyperparameter you tune.

Is knowledge distillation the same as fine-tuning?

No. Fine-tuning adapts an existing model to new data. Distillation trains a (usually smaller) model to imitate another model. In practice LLM distillation is often done by fine-tuning a small model on text generated by a large one, which is why the terms get mixed up.

How do I implement knowledge distillation in PyTorch?

Compute the teacher’s logits with torch.no_grad(), then train the student on alpha * KL(log_softmax(student / T), softmax(teacher / T)) * T * T + (1 - alpha) * cross_entropy(student, labels) using F.kl_div(..., reduction="batchmean"). The runnable example above shows the full loop on scikit-learn’s digits dataset.

Is knowledge distillation better than quantization?

They solve different problems and work well together. Quantization keeps the architecture and stores weights in fewer bits, with no or little training. Distillation produces a genuinely smaller architecture but needs a training run. A common pipeline is to distill and then quantize the student.


Final Thoughts

Knowledge distillation bridges the gap between large, powerful models and real-world deployment. By making AI both smarter and leaner, this technique is transforming applications from edge devices to real-time systems. Next time you use a quick AI assistant on your phone, think about the distilled knowledge powering it. And when your student model is ready to serve behind an API, my Ray Serve guide covers model serving with autoscaling and request batching. Stay tuned for more insights tomorrow!

Thanks for reading my blog post! Feel free to check out my other posts or contact me via the social links in the footer.

Keep reading

All posts