Why do knowledge distillation?
Knowledge distillation transfers capabilities from a large, complex model (teacher) to a smaller, simpler one (student). It enables deploying performant models where computational resources or latency constraints prevent using the teacher directly.
Core answer
Answer
The process allows the student model to learn not only from ground truth labels but also from the teacher's softer output probabilities and potentially intermediate representations. Key benefits include achieving higher performance than training the small model solely on labels, significantly reducing model size and inference costs, and speeding up prediction. This is crucial when deploying to edge devices or under strict runtime requirements.
Its primary application value lies in model compression for deployment on resource-constrained devices like mobile phones and embedded systems. It also improves the performance of small models acting independently and reduces cloud inference costs by replacing large models with effective, compact substitutes after distillation. This translates to wider accessibility of advanced AI capabilities.
Related keywords