sergiopaniego/qwen3-0.6b-pimono-gkd-lr2e5
The sergiopaniego/qwen3-0.6b-pimono-gkd-lr2e5 model is a fine-tuned version of the Qwen3-0.6B architecture, developed by sergiopaniego. This 0.8 billion parameter model, with a 32768 token context length, was trained using the GKD (On-Policy Distillation of Language Models) method on the pi-mono-chat dataset. It is optimized for generating conversational text based on its specialized training, making it suitable for chat-based applications.
Loading preview...
Model Overview
This model, sergiopaniego/qwen3-0.6b-pimono-gkd-lr2e5, is a specialized fine-tune of the Qwen3-0.6B base model. Developed by sergiopaniego, it leverages a 0.8 billion parameter architecture with a substantial 32768 token context length.
Key Training Details
The model was fine-tuned on the sergiopaniego/pi-mono-chat dataset using the TRL (Transformers Reinforcement Learning) framework. A notable aspect of its training is the application of GKD (On-Policy Distillation of Language Models), a method designed for learning from self-generated mistakes, as introduced in the paper "On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes" (ICLR 2024).
Use Cases
This model is particularly well-suited for:
- Generating conversational responses in chat-based applications.
- Exploring the effects of GKD training on smaller language models.
- Applications requiring a model fine-tuned on specific chat datasets.