LLM Distillation Is Function Approximation, Not Model Copying

Knowledge distillation is often described as transferring knowledge from a large teacher model into a smaller student model.

That metaphor is useful until it becomes too literal.

The student does not receive the teacher's parameters, hidden causal structure, or some compressed essence of everything the teacher knows. It receives training signals generated by the teacher under a chosen data distribution.

Distillation is therefore a function-approximation problem.

Let the teacher define

and the student define

Distillation chooses a distribution over inputs $P_D(x)$ and an objective that makes the student approximate selected aspects of $f_T$ on that distribution.

The central limitation follows immediately:

Everything depends on what the teacher is asked, what outputs are recorded, and what loss the student optimizes.

Classical distillation uses soft targets

Suppose a teacher and student predict over vocabulary $\mathcal V$.

For token $v$, teacher and student logits are

and

With temperature $\tau>0$, softened probabilities are

and similarly for the student.

A common distillation loss minimizes

The factor $\tau^2$ compensates for gradient scaling under temperature changes in the classical formulation.

The teacher provides more information than a single hard target.

If the next token is "Paris" but the teacher also gives moderate probability to "Lyon" and very low probability to unrelated words, the student sees the teacher's similarity structure among alternatives.

That is the original attraction of soft-target distillation.

Temperature controls how much dark knowledge is exposed

When

the softmax is unchanged.

When

the probability distribution becomes flatter.

This exposes relative probabilities among non-maximal classes.

As

differences shrink and the distribution approaches uniformity.

Very high temperature can therefore erase useful structure.

Very low temperature approaches a hard target.

Temperature is a model-selection parameter, not a ritual constant.

Hard labels and soft targets optimize different information

Suppose the teacher distribution is

A hard target keeps only

The student loses information about which alternatives the teacher considered plausible.

Soft targets preserve that ranking.

This can improve generalization when the teacher's probability structure contains useful information beyond the argmax.

The benefit depends on calibration and teacher quality.

A confidently wrong teacher can transfer its error with great efficiency.

Language models make full-logit distillation expensive

For a vocabulary of size $V$ and sequence length $T$, storing all teacher logits requires roughly

numbers per example.

With vocabularies of tens or hundreds of thousands of tokens, this becomes expensive.

Black-box APIs often do not expose full logits anyway.

This is one reason LLM distillation frequently shifts from white-box logit matching to sequence-level supervision.

Sequence-level distillation trains on teacher generations

Instead of storing token distributions, ask the teacher to generate a response

The student then minimizes a standard autoregressive loss:

This turns the teacher into a synthetic-data generator.

Operationally, it resembles SFT.

Conceptually, the labels are teacher-generated rather than human-authored.

Sequence distillation throws away uncertainty

If the teacher can produce several plausible answers, one sampled sequence reveals only one trajectory.

The student does not see the alternatives that were possible.

If decoding is greedy, the dataset may become even narrower.

Sequence-level distillation therefore trades information richness for simplicity.

It is easy to store and works with black-box teachers.

It also compresses the teacher distribution into selected outputs.

Decoding policy becomes part of the training data generator

Teacher outputs depend on:

• temperature,
• top-$p$,
• top-$k$,
• beam search,
• stop criteria,
• system prompt.

Therefore the synthetic dataset is drawn from

where $\phi$ denotes the decoding configuration.

Changing $\phi$ changes the labels.

A distilled student is partly a model of the teacher's decoding policy, not just its underlying logits.

Greedy teachers can create low-diversity students

If teacher generation uses deterministic decoding, repeated prompts can yield highly stereotyped outputs.

The student can learn those surface regularities.

This may reduce diversity even when the original teacher was capable of many valid responses.

A useful design can sample multiple teacher responses per prompt:

The student then sees a broader approximation to the teacher's conditional support.

Synthetic data quality is bounded by the teacher

Teacher-generated data can scale rapidly.

It also reproduces teacher errors.

If

the synthetic corpus contains systematic label noise.

That noise may be concentrated in precisely the hard regions one hopes the student will learn.

Teacher confidence or self-consistency can help triage examples, but they are not guarantees of correctness.

Distillation can amplify teacher biases

Suppose the teacher systematically prefers:

• verbose answers,
• one reasoning template,
• one coding style,
• one dialect,
• one citation format.

A student trained mostly on teacher-generated outputs can inherit these preferences more strongly than the original teacher, because the dataset contains a filtered subset of the teacher's output distribution.

Compression can therefore sharpen artefacts.

Rationales provide an extra supervision channel

Instead of training only on the final answer, a teacher can generate a rationale:

and answer

The student trains on

This can be useful because the intermediate text provides more tokens associated with the task structure.

Work on rationale distillation has shown that smaller models can benefit from such supervision on some tasks. citeturn1search1

The safe interpretation is that rationales provide additional generated training signal.

They are not verified transcripts of an internal reasoning process.

Teacher rationales can be post hoc

A language model can produce an explanation after choosing an answer.

The explanation may be plausible without being causally responsible for the answer.

Therefore

should be treated as generated text.

If rationale correctness matters, evaluate it independently.

A beautiful explanation for a wrong answer is still wrong supervision.

Distillation can use task decompositions without pretending they are hidden thoughts

A safer approach is to ask the teacher for externally checkable intermediate artefacts:

• equations,
• tool calls,
• retrieved evidence,
• intermediate labels,
• structured plans.

These objects can be validated.

The student can then learn a decomposed task without relying on unverifiable free-form rationales.

Capacity mismatch places a hard limit on transfer

Let the teacher hypothesis class be

and the student class be

If

the student cannot reproduce the teacher exactly.

The best achievable approximation error is

A tiny student may simply lack the capacity required for some teacher behaviours.

No distillation objective can eliminate that representational gap.

Distillation performance depends on the input distribution

Suppose distillation prompts come from

but deployment uses

A student may approximate the teacher well on $P_D$ and poorly elsewhere.

This is covariate shift.

Teacher-student agreement should therefore be evaluated on deployment-like prompts not used to generate training data.

Off-policy sequence distillation creates exposure bias

In ordinary sequence distillation, the student trains on prefixes generated by the teacher.

At deployment, it conditions on prefixes generated by itself.

These state distributions differ.

Let

be a teacher-generated prefix and

a student-generated prefix.

Training mostly observes

while inference visits

Errors can compound when the student enters states never seen during training.

On-policy distillation addresses student-state coverage

On-policy distillation generates trajectories from the student and asks the teacher for supervision on those states.

The training distribution becomes closer to the student's own inference distribution.

Conceptually:

then optimize the student toward teacher targets conditioned on $h_t$.

Recent work argues that this can reduce the train-inference mismatch inherent in purely teacher-generated trajectories. citeturn0search3

The cost is additional teacher inference during training.

Forward KL and reverse KL behave differently

Standard logit distillation often minimizes

This strongly penalizes the student for assigning too little mass where the teacher assigns probability.

It tends to be mode-covering.

Reverse KL,

penalizes student probability mass placed where the teacher has little support and can be more mode-seeking.

MiniLLM explored reverse-KL-style objectives for generative model distillation to address difficulties with standard maximum-likelihood sequence distillation. citeturn1search0

The divergence determines which approximation errors are expensive.

Distillation loss should match deployment behaviour

If the deployment requires diverse generation, an objective that aggressively collapses modes can be harmful.

If the deployment requires one deterministic structured answer, mode-seeking behaviour may be acceptable.

There is no universally correct divergence independent of the application.

A hybrid loss can combine ground truth and teacher supervision

When human labels exist, do not necessarily discard them.

One can optimize

This prevents the teacher from becoming the only source of truth.

The mixture weight $\lambda$ should be validated.

Teacher selection is a modelling decision

The largest available teacher is not automatically the best teacher.

A useful teacher should be:

• strong on the target tasks,
• stable under the prompt format,
• calibrated enough for soft targets if logits are used,
• affordable to query,
• legally usable for synthetic-data generation.

A highly capable general teacher can still be poor on a specialized domain.

Ensembles can distill consensus

If several teachers are available, aggregate their outputs.

For logit distillation, one can average distributions:

For sequence-level distillation, multiple teachers can generate candidates that are filtered or ranked.

Teacher disagreement is itself informative.

If teachers disagree strongly, the example may be ambiguous or out of distribution.

Distillation can leak proprietary teacher behaviour

Black-box distillation raises governance questions.

A synthetic dataset generated from a proprietary model can encode:

• stylistic signatures,
• copyrighted memorized content,
• sensitive outputs,
• policy behaviour.

The fact that the student is smaller does not remove provenance concerns.

Training-data lineage should include teacher model, version, prompts and generation settings.

Distillation is not guaranteed to preserve safety behaviour

A teacher may refuse some prompts correctly.

If the synthetic dataset contains mostly benign tasks, the student sees little refusal behaviour.

The distilled model can therefore regress on safety even while matching task performance.

Include protected behaviours explicitly in evaluation and, if needed, in the training distribution.

Distillation and quantization solve different deployment problems

Quantization keeps the same architecture and changes numerical representation.

Distillation changes the architecture or parameter count by training a new student.

They can be combined:

The order matters.

Quantizing the teacher during data generation may alter the supervision signal.

Quantizing the student after distillation adds another approximation layer.

Evaluate each stage.

Distillation can also reduce inference depth or width

A student may have:

• fewer layers,
• smaller hidden width,
• fewer attention heads,
• smaller vocabulary embeddings,
• a different architecture.

This can produce speedups that weight-only quantization cannot because the actual computation graph is smaller.

That is one of distillation's strongest advantages.

Smaller parameter count does not guarantee lower latency

Runtime performance still depends on kernels and hardware.

A dense 3B model can sometimes run more efficiently than an awkwardly shaped 2B model.

Benchmark the actual student.

Distillation should report compression and quality together

Let teacher size be

and student size

Compression ratio is

Quality retention for metric $S$ can be summarized as

when the metric scale permits ratios.

The interesting object is the trade-off between $C$ and $R$.

A student that preserves 99% of quality at 2x compression and one that preserves 90% at 10x compression solve different deployment problems.

Teacher-student agreement is not enough

A student can agree with the teacher and both can be wrong.

Evaluation must include external ground truth or human judgement where possible.

Measure:

Agreement $A$ is diagnostic.

Task score $S$ is the outcome.

Student performance can exceed the teacher on a narrow task

This is not paradoxical.

If distillation data focus on one task distribution and the student receives additional human labels or regularization, it can outperform the teacher on that narrow benchmark.

The student is not a literal copy.

It is a separately trained model using teacher supervision as one data source.

Self-distillation is still distillation

A model can act as its own teacher across checkpoints or decoding procedures.

For example:

and train a later student initialized from

Self-distillation can regularize or consolidate predictions.

It does not require the teacher architecture to be larger.

The defining property is the use of teacher-generated targets.

Distillation datasets should preserve prompt provenance

For each synthetic example, log:

• source prompt ID,
• teacher model and version,
• system prompt,
• decoding configuration,
• timestamp,
• teacher output,
• filtering decision,
• human corrections if any.

Without provenance, synthetic datasets become impossible to audit.

Filtering changes the target distribution

Suppose only teacher answers that pass a validator are retained.

Then the student trains on

not the teacher's unconditional output distribution.

This may be desirable.

It should be acknowledged.

Filtering can substantially improve synthetic data, but it creates selection bias.

Confidence filtering can overconcentrate easy examples

If only high-confidence teacher outputs are retained, the dataset can become dominated by easy prompts.

Hard prompts disappear.

The student then looks good on training loss but fails precisely where teacher confidence was low.

A better design can stratify by difficulty and retain some hard examples for evaluation or targeted supervision.

Distillation needs a clean test set

Do not evaluate on prompts used to generate teacher training data.

Maintain a held-out prompt set sampled independently from deployment-like sources.

Compare:

This reveals both teacher gap and distillation gain.

A practical sequence-distillation workflow is simple

Generate teacher outputs into JSONL:

1
2
3
4
5
6
{
  "messages": [
    {"role": "user", "content": "Explain ..."},
    {"role": "assistant", "content": "Teacher response ..."}
  ]
}

Then train the student with ordinary SFT.

The implementation is easy.

The hard part is the synthetic-data design.

Soft-target distillation requires teacher logits

If the teacher is locally accessible, store top-$k$ logits or full distributions.

A memory-saving approximation stores only the largest teacher probabilities:

Renormalize carefully if the omitted probability mass matters.

Top-$k$ logit compression changes the teacher distribution.

Distillation can be multi-stage

A useful pipeline may be:

Intermediate students can make later distillation cheaper.

But each stage adds approximation error.

Measure cumulative degradation.

Distillation objectives should be ablated

At minimum, compare:

If logits are available, also compare soft-target distillation.

If rationales are added, include a no-rationale baseline.

Otherwise one cannot know which supervision channel caused the improvement.

Release criteria should include efficiency

A student exists to change the quality-resource frontier.

Measure:

• task quality,
• memory,
• latency,
• throughput,
• energy or cost,
• model load time.

A student that is slightly smaller but slower on the target hardware has failed the systems objective.

The real target is the Pareto frontier

Let

be utility and

deployment cost.

A distilled student is useful if it occupies a better point in

space.

The goal is not to maximize teacher imitation.

It is to retain enough useful behaviour at substantially lower cost.

Distillation is an approximation problem with measurement error

Teacher outputs are measurements of teacher behaviour.

They can be noisy, biased, incomplete and distribution-dependent.

The student then approximates those measurements under its own capacity constraints.

The full chain is

Every arrow can introduce error.

This is why "distill the model" is not a complete method specification.

References

Beyer, L., Zhai, X., Royer, A., Markeeva, L., Anil, R., & Kolesnikov, A. (2022). Knowledge Distillation: A Good Teacher Is Patient and Consistent. CVPR 2022.

Gu, Y., Dong, L., Wei, F., & Huang, M. (2024). MiniLLM: Knowledge Distillation of Large Language Models. ICLR 2024.

Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the Knowledge in a Neural Network. NIPS Deep Learning Workshop.

Kim, Y., & Rush, A. M. (2016). Sequence-Level Knowledge Distillation. EMNLP 2016.

Magister, L. C., Mallinson, J., Adamek, J., Malmi, E., & Severyn, A. (2023). Teaching Small Language Models to Reason. ACL 2023.

Shridhar, K., Stolfo, A., & Sachan, M. (2023). Distilling Reasoning Capabilities into Smaller Language Models. Findings of ACL 2023.

Recent on-policy distillation work for language models. Accessed 21 September 2026.