Iterative Knowledge Distillation: Teacher-Student LLM Framework for Response Generation and Adaptive Training — 29p — Nagraj Naidu, Laxmi Manasa Gorugant

Efficient knowledge transfer between large language models (LLMs) remains a critical challenge in machine learning due to the computational complexity and resource demands involved in training these models. This research presents an iterative teacher-student knowledge distillation framework wherein both the teacher and student models are transformer-based LLMs. The teacher model generates responses to a predefined set of requests, which serve as training data for the student model in a supervised learning setup.

This iterative approach facilitates the continuous refinement of the student’s performance by leveraging the expertise of the teacher model. To optimize the training process, we employ the Optuna framework to dynamically fine-tune hyperparameters. It is expected that the proposed framework will significantly enhance the student’s accuracy and generalization capabilities compared to conventional distillation methods.

The anticipated findings of this research underscore the potential of adaptive, metric-driven training loops in optimizing knowledge distillation processes. This study aims to contribute to the broader understanding of knowledge transfer mechanisms in LLMs and offers a robust framework for developing high-performance student models that closely approximate the performance of their teacher models. The implications of this work are expected to be significant, highlighting avenues for future research in adaptive learning systems and the refinement of distillation techniques.

Dakota State University
Mark Spanier