What does knowledge distillation mean?
Knowledge distillation refers to a model compression technique in which a smaller "student" model is trained to mimic the behavior of a larger, more complex "teacher" model or ensemble of models. Its primary goal is to transfer the knowledge captured by the cumbersome teacher into a student model that is more efficient to deploy.
Core answer
Answer
This process typically involves training the student model not only on the standard hard labels (e.g., correct class answers like "cat"), but more importantly, on the "soft labels" output by the teacher – the probability distributions over classes generated by the teacher's softmax layer (e.g., high probability for "cat", medium for "lynx", low for "tiger"). These soft labels contain richer information about the teacher's learned representations and relationships between classes. Knowledge distillation can involve transferring different types of "knowledge", such as the output distribution (most common), intermediate feature representations, or relational information between data points.
Knowledge distillation enables the deployment of high-performance deep learning models on resource-constrained edge devices or mobile applications where the original teacher model is too large or computationally heavy. It achieves model compression, reduces inference latency and memory footprint, and sometimes even improves generalization by acting as a form of regularization. It is widely used to create smaller, faster models suitable for production environments without significant loss in accuracy.
Related keywords