Model Distillation

Model distillation is a technique for transferring behavior from a teacher model to a student model. The teacher is typically a larger, more capable model, while the student is typically smaller and less expensive to serve. Rather than copying the teacher’s parameters, the student learns from signals produced by the teacher. These can include generated answers, probability distributions over possible outputs, or intermediate representations when the training setup permits access to them. For foundation models accessed through APIs, training on teacher-generated prompt-and-response examples is a practical approach.

A cloud implementation starts with a representative set of application inputs. Engineers ask the teacher to produce outputs, filter or review those outputs, and use the resulting examples to train an appropriate student model. Training data, model artifacts, and evaluation results can be stored in object storage, with managed training infrastructure handling compute and checkpoints. The student is then deployed through a managed model endpoint or compatible inference platform. Available teacher–student combinations, training methods, and deployment options depend on the platform and model licenses.

Consider a support application that uses a large foundation model to classify tickets and produce short summaries. If these tasks have stable formats and limited scope, a smaller student may learn to perform them adequately. The application can serve routine requests with the student and route difficult cases to the teacher. Evaluation should use held-out examples that reflect production traffic, including unusual inputs, safety-sensitive requests, and cases requiring abstention. Teacher agreement alone is insufficient because the teacher can also be wrong.

The main economic trade-off is upfront investment versus recurring savings. Distillation incurs teacher inference charges for generating examples, data preparation and review costs, student training charges, evaluation effort, and ongoing maintenance. A smaller student may reduce inference charges, accelerator memory requirements, and latency, but those benefits must be measured on the actual hosting configuration. If setup costs are $6,000 and net serving savings are $0.003 per request, the simplified break-even point is two million requests. That calculation excludes additional fixed hosting, retraining, and fallback costs unless they are included in the estimates.

Compared with pre-training a foundation model from scratch, distillation commonly requires much less compute because it usually adapts an existing student to selected teacher behavior. Fine-tuning describes a training procedure, whereas distillation describes the source and purpose of the learning signal; fine-tuning a student on teacher-generated examples can therefore be both. Fine-tuning on human-labeled examples is not automatically distillation. Neither method guarantees that a smaller model will retain the teacher’s broad capabilities.

In-context learning supplies instructions or examples at inference time without updating model weights. It avoids a student-training job but consumes prompt tokens repeatedly and remains subject to context-window limits. Retrieval-augmented generation, or RAG, retrieves external information for each request, adding retrieval infrastructure and context-processing costs while supporting access to changing information. Distillation changes model behavior through training and does not replace a current knowledge source. A distilled student can still use RAG, and the combined system should be evaluated end to end.

Distillation is not equivalent to quantization or pruning, which reduce numerical precision or remove model components. These techniques can be combined with distillation, but they have different mechanisms and quality trade-offs. Distillation also does not eliminate hallucinations, guarantee faster inference, or automatically remove sensitive information from generated training data. Engineers should check model-use rights, protect training data, measure task-specific quality and latency, and repeat evaluation when production inputs or requirements change.