Distillation Techniques for Compressing LLMs into Smaller Student Models

Large language models have achieved remarkable capabilities, but their size presents a fundamental deployment challenge. A model like GPT-3 with 175 billion parameters requires hundreds of gigabytes of memory and powerful GPU clusters to run, making it impractical for most real-world applications. Even smaller models with 7-13 billion parameters strain typical hardware resources and deliver unacceptable latency for interactive use cases. This size-capability tension creates a critical need: can we compress these large “teacher” models into much smaller “student” models while preserving most of their capabilities?

Knowledge distillation offers a powerful solution to this compression challenge. Rather than training small models from scratch—where they struggle to match large models’ performance—distillation trains student models to mimic teacher models’ behavior. The student learns not just from labeled data but from the teacher’s rich probability distributions, capturing nuances that hard labels alone cannot convey. When done effectively, distillation can compress models by 10-100x while retaining 95-98% of the original performance, making sophisticated language understanding and generation accessible on consumer hardware, mobile devices, and resource-constrained environments.

The Foundation: Why Distillation Works

Understanding why distillation succeeds requires examining what student models learn from teachers beyond what they’d learn from training data alone.

Soft targets and probability distributions:

Standard supervised learning trains models on hard labels—for a classification task, the target is a one-hot vector indicating the correct class. This provides minimal information: one class gets probability 1.0, all others get 0.0. The model has no information about which wrong answers are “more wrong” or which classes are similar to the correct one.

Teacher models produce soft probability distributions across all possible outputs. For a language model predicting the next token, the teacher might assign 0.4 probability to the most likely token, 0.2 to another plausible token, 0.15 to a third, and smaller probabilities to thousands of other tokens. These soft distributions contain rich information about token relationships, contextual appropriateness, and semantic similarity that hard labels obliterate.

When students train on these soft distributions, they learn the teacher’s nuanced understanding. If the teacher assigns similar probabilities to synonyms or related concepts, the student learns these relationships. If certain tokens are contextually implausible, their near-zero probabilities teach the student to avoid them.

Temperature scaling and softening distributions:

A key distillation technique is temperature scaling, which softens the teacher’s probability distributions to make them more informative. Standard softmax converts logits to probabilities, but adding a temperature parameter T controls the distribution’s “peakiness”:

p(i) = exp(logit(i) / T) / Σ exp(logit(j) / T)

With T=1 (standard), high-probability tokens dominate. With T>1 (typical values: 2-5), probabilities become more uniform, revealing the teacher’s relative rankings even for low-probability options. A token with probability 0.01 at T=1 might have 0.05 at T=3, providing much stronger learning signal to the student.

During distillation, both teacher and student use the same temperature T when computing distributions, then train the student to match these softened distributions. At inference, the student uses T=1, producing sharp probability distributions suitable for actual predictions.

Dark knowledge and learned biases:

The teacher’s training has encoded “dark knowledge”—implicit understanding about task structure, language patterns, and concept relationships that never appears in explicit outputs. A teacher trained on billions of tokens has learned subtle grammatical constraints, semantic associations, and pragmatic conventions that hard labels cannot capture.

Distillation transfers this dark knowledge through probability distributions. When the teacher assigns tiny but non-zero probabilities to incorrect but plausible options, it’s teaching about language structure. When it spreads probability across synonyms, it’s encoding semantic similarity. The student absorbs these patterns, gaining understanding that would require vastly more training data to learn from scratch.

Loss Function Design for Effective Distillation

The loss function determines what the student optimizes for during training, critically affecting the quality of the compressed model.

Distillation loss components:

Effective distillation typically combines two loss terms:

Knowledge distillation loss (L_KD): Measures how well the student’s soft probability distribution matches the teacher’s soft distribution. Kullback-Leibler (KL) divergence is the standard metric:

L_KD = KL(Teacher_soft || Student_soft)

This pushes the student to replicate the teacher’s nuanced probability distributions across all outputs.

Task-specific loss (L_task): Measures performance on the actual task using hard labels. For language models, this is typically cross-entropy loss against the ground-truth next token. For classification, it’s standard classification loss against true labels.

The total loss combines both with a weighting hyperparameter α:

L_total = α * L_KD + (1 - α) * L_task

Typical α values range from 0.5 to 0.9, heavily weighting distillation loss while maintaining some direct supervision from ground truth.

Balancing distillation and task objectives:

The balance between distillation and task losses affects what the student learns:

High α (0.8-0.95): Student closely mimics teacher behavior, potentially including teacher biases and errors. This maximizes capability transfer but also transfers weaknesses. Best when teacher quality is high and you want maximum preservation of teacher capabilities.

Moderate α (0.5-0.7): Balances teacher guidance with independent learning from data. Student may correct some teacher errors while still benefiting from teacher knowledge. Good default choice for most scenarios.

Low α (0.2-0.4): Student relies more on ground truth supervision, using teacher primarily as regularization. Appropriate when teacher quality is questionable or when you want the student to potentially surpass the teacher on specific metrics.

Feature-based distillation:

Beyond matching output distributions, feature-based distillation matches intermediate representations. Student hidden states are trained to match teacher hidden states at corresponding layers through additional loss terms:

L_feature = Σ MSE(Student_hidden[i], Teacher_hidden[j])

This provides denser supervision—the student learns not just final outputs but intermediate processing steps. However, it requires careful layer matching (student and teacher rarely have the same architecture) and adds training complexity.

Feature distillation is particularly effective for very aggressive compression (100x+) where output-only distillation struggles to transfer sufficient knowledge.

Training Strategies and Data Selection

How you conduct distillation training significantly impacts the final student model’s quality.

Data requirements for distillation:

Distillation doesn’t require labeled data in the traditional sense—the teacher generates labels through its predictions. However, unlabeled input data is still necessary:

In-domain data: Text from the target domain helps the student learn relevant patterns. For a medical language model, medical texts are ideal. For general language understanding, diverse web text works well.

Scale vs. quality: Distillation can work with less data than training from scratch. While a teacher might train on 1 trillion tokens, effective distillation often succeeds with 10-100 billion tokens—still substantial but more manageable. Quality matters more than quantity—diverse, representative data produces better students than massive but repetitive datasets.

Synthetic data generation: The teacher can generate synthetic training data. Provide diverse prompts and have the teacher generate continuations. This synthetic data, generated by the teacher, can be excellent training material as it reflects the teacher’s knowledge directly.

Multi-task distillation:

Rather than distilling on a single task, multi-task distillation exposes the student to diverse tasks simultaneously. Present the student with translation, summarization, question answering, and other tasks, distilling from the teacher’s performance across all of them.

This multi-task approach:

• Improves generalization by exposing diverse patterns
• Helps preserve broad capabilities during compression
• Prevents overfitting to any single task’s peculiarities
• Creates more versatile compressed models

However, it requires carefully balancing task sampling to prevent the model from focusing excessively on high-resource tasks while neglecting others.

Progressive distillation strategies:

Instead of compressing directly from a very large teacher to a tiny student (e.g., 175B → 1B), progressive distillation uses intermediate models:

1. Distill 175B teacher → 30B intermediate student
2. Distill 30B intermediate → 7B smaller student
3. Distill 7B smaller → 1B final student

Each step compresses by a more manageable ratio (5-7x), allowing better knowledge transfer. The intermediate models learn effectively from their respective teachers, and the final student benefits from multiple distillation stages.

This progressive approach is more expensive (requires training multiple models) but often produces superior final students, particularly for very aggressive compression ratios.

Architecture Considerations for Student Models

The student model architecture significantly impacts how effectively it can absorb teacher knowledge.

Capacity and architectural choices:

Students need sufficient capacity to approximate teacher behavior but not so much capacity that compression benefits disappear. Typical compression ratios:

Conservative (3-5x): Student has 20-30% of teacher parameters. Retains 95-98% of teacher performance. Still relatively large but more deployable.

Moderate (10-20x): Student has 5-10% of teacher parameters. Retains 90-95% of performance. Sweet spot for many applications—substantial deployment benefits with acceptable quality trade-offs.

Aggressive (50-100x+): Student has 1-2% of teacher parameters. Retains 80-90% of performance. Enables mobile/edge deployment but requires accepting significant capability loss.

Architectural similarity matters:

Students architecturally similar to teachers (e.g., both transformers with similar layer structures) distill more easily. Mapping teacher outputs to student inputs is straightforward, and students can more directly replicate teacher computation patterns.

Architecturally different students (e.g., transformer teacher → RNN student, or different attention mechanisms) face challenges. Feature-based distillation becomes harder when intermediate representations differ structurally. However, architectural differences can provide benefits like improved inference speed or reduced memory beyond parameter reduction alone.

Depth vs. width trade-offs:

When designing smaller student architectures, you can reduce depth (fewer layers), width (fewer dimensions per layer), or both. These choices have different implications:

Reducing depth: Fewer layers mean shorter computation paths and faster inference. However, deep networks often capture hierarchical abstractions better. Students with fewer layers may struggle to replicate teacher’s multi-level reasoning.

Reducing width: Narrower layers (fewer dimensions) reduce parameters while maintaining depth. This preserves hierarchical processing but constrains each layer’s representational capacity. Can work well if the narrower layers have sufficient capacity for the task.

Balanced reduction: Proportionally reducing both depth and width often works best, maintaining the architectural ratios that made the teacher effective while scaling down overall size.

Advanced Distillation Techniques for LLMs

Recent research has developed specialized distillation techniques particularly effective for large language models.

Selective distillation and targeted knowledge transfer:

Not all teacher knowledge is equally valuable to transfer. Selective distillation identifies and prioritizes critical knowledge:

Task-specific distillation: Focus distillation on tasks the compressed model will actually perform. If deploying for summarization, heavily weight summarization examples during distillation even if the teacher has broad capabilities.

Difficulty-based sampling: Emphasize examples where the teacher demonstrates sophisticated reasoning or where naive models fail. These hard examples contain the most valuable knowledge to transfer.

Confidence-based weighting: Weight distillation loss by teacher confidence. High-confidence teacher predictions contain clearer signals; low-confidence predictions may transfer uncertainty that wastes student capacity.

Layer-wise distillation with attention mechanisms:

Modern transformer models contain 20-100 layers. Rather than treating all layers equally, layer-wise distillation strategically matches student and teacher layers:

Last-layer matching: Match student’s final layer to teacher’s final layer. This captures output-level knowledge without requiring architectural similarity throughout the network.

Selected-layer matching: Match specific student layers to carefully chosen teacher layers. For a 12-layer student learning from a 48-layer teacher, you might match student layers 3, 6, 9, 12 to teacher layers 12, 24, 36, 48. This provides supervision at multiple depths without requiring every layer to match.

Attention distribution distillation: Beyond hidden states, match attention distributions. Teachers learn which tokens to attend to in self-attention. Transferring these attention patterns helps students focus on relevant context, improving comprehension and generation quality.

Quantization-aware distillation:

Many deployments quantize models to lower precision (INT8, INT4) for further compression and speed. Quantization-aware distillation incorporates quantization into the distillation process:

Train the student with quantization operations in the forward pass, so distillation learns to match the teacher while accounting for quantization noise. This produces students that both compress architecturally (fewer parameters) and numerically (lower precision) without compounding accuracy losses.

The result is dramatically smaller models—a 7B parameter model distilled to 1B parameters and then quantized to INT4 might require only 500MB of memory versus 14GB for the original, enabling deployment on smartphones and edge devices.

Evaluation and Quality Assurance

Assessing distilled models requires careful evaluation beyond simple accuracy metrics to ensure capability preservation and identify failure modes.

Comprehensive evaluation frameworks:

Distilled LLMs require evaluation across diverse dimensions:

Task performance: Standard benchmarks (GLUE, SuperGLUE, etc.) measure performance on specific tasks. Student models should retain 90-98% of teacher performance depending on compression ratio.

Generalization: Test on out-of-distribution data to verify the student learned generalizable patterns, not just memorized teacher quirks on training data.

Capability coverage: Evaluate across the full range of teacher capabilities—reasoning, knowledge retrieval, instruction following, multi-step tasks. Distillation sometimes drops specific capabilities even when aggregate metrics look good.

Calibration: Check whether student probabilities are well-calibrated. Poorly calibrated students might achieve good accuracy but generate overconfident or underconfident probability estimates, problematic for downstream use.

Identifying failure modes:

Distillation can fail in specific ways that require attention:

Capability collapse: Student performs well on common patterns but completely fails on rare but important cases. For example, handling code, math, or specialized terminology might collapse despite good average performance.

Bias amplification: Distillation can amplify teacher biases. If the teacher exhibits subtle biases, the student might exaggerate them, particularly under aggressive compression where nuance is lost.

Catastrophic forgetting: When distilling multi-task models, students sometimes excel on some tasks while catastrophically forgetting others. Multi-task evaluation catches this, but it requires careful monitoring.

Reasoning degradation: Students might retain factual knowledge but lose multi-step reasoning abilities. Chain-of-thought evaluation reveals whether the student can still perform complex reasoning or merely memorizes common reasoning patterns.

Iterative refinement:

Distillation is rarely a one-shot process. Effective distillation involves iteration:

1. Initial distillation with standard techniques
2. Evaluate comprehensively, identifying weak areas
3. Targeted distillation focusing on identified weaknesses—oversample examples from weak areas, adjust loss weightings, or use curriculum learning
4. Re-evaluate and iterate until acceptable performance across all critical dimensions

This iterative refinement is essential for production deployments where specific capabilities matter more than aggregate benchmark scores.

Deployment Considerations and Practical Trade-offs

Successfully deploying distilled models requires understanding practical constraints beyond just model size.

Latency vs. throughput:

Smaller models generally have lower latency (faster per-request response) and higher throughput (more requests per second). However, the relationship isn’t always linear:

A 10x parameter reduction might give 3-5x latency improvement due to memory bandwidth constraints, fixed overhead in model serving infrastructure, and diminishing returns in very small models where overhead dominates.

Profile your specific deployment environment to measure actual latency/throughput gains rather than assuming linear scaling with parameter count.

Memory hierarchy effects:

Modern deployment involves complex memory hierarchies—GPU HBM, CPU RAM, SSD storage. Model size determines where the model resides:

• < 1GB: Fits entirely in GPU memory on consumer GPUs
• 1-10GB: Requires good GPUs but accessible on modern hardware
• 10-50GB: Requires professional GPUs or CPU inference with slower performance
• > 50GB: Requires multi-GPU or specialized serving infrastructure

These memory tiers create discontinuous deployment cost functions. A 12GB model and 8GB model have dramatically different deployment options, even though the size difference seems modest.

Edge and mobile deployment:

Aggressive distillation (50-100x compression) enables deployment on mobile devices and edge hardware. Considerations include:

Mobile constraints: iOS/Android apps have strict size limits (100-200MB). Aggressive quantization combined with distillation can fit capable LLMs in these constraints.

Inference frameworks: TensorFlow Lite, ONNX Runtime Mobile, and other frameworks optimize distilled models for mobile inference. Some distillation tools integrate with these frameworks for end-to-end optimization.

Battery and thermal: Small models not only fit in memory but generate less heat and drain batteries slower. For continuous inference applications (like voice assistants), power efficiency from distillation is as important as size reduction.

Conclusion

Knowledge distillation has matured into an essential technique for making large language models practical for real-world deployment, enabling compression ratios of 10-100x while retaining 80-95% of teacher model capabilities through careful transfer of soft probability distributions and dark knowledge. By training students to match teacher output distributions rather than just hard labels, incorporating multiple loss components including feature-based and attention-based distillation, and employing progressive distillation strategies for aggressive compression, practitioners can create deployable models that preserve most of the sophisticated language understanding and generation capabilities that made large models impressive. The key lies in matching distillation techniques to compression goals—standard output distillation for modest compression, multi-task and feature-based distillation for moderate compression, and progressive distillation with quantization-aware training for aggressive compression to mobile-scale models.

The practical impact of effective distillation extends far beyond just creating smaller models. It democratizes access to advanced language AI by making models deployable on consumer hardware, enables edge and mobile applications that simply cannot send data to cloud services for inference, reduces operational costs for serving language models at scale by orders of magnitude, and provides privacy benefits by enabling on-device processing. As techniques continue advancing with better layer-matching strategies, improved synthetic data generation, and distillation methods specifically designed for emerging LLM architectures, the gap between distilled student performance and teacher performance continues narrowing, making it increasingly feasible to deploy highly capable language models in resource-constrained environments without accepting the dramatic capability losses that once made compression impractical.