metacognitive-behavioral-tuning/Qwen3-0.6B-gpt-oss-distill
The metacognitive-behavioral-tuning/Qwen3-0.6B-gpt-oss-distill is a 0.8 billion parameter causal language model based on Qwen3-0.6B, developed by metacognitive-behavioral-tuning. This model is specifically distilled from a larger teacher model (gpt-oss-120b) using SFT and GRPO methods. It is optimized for multi-hop question answering tasks, demonstrating performance on benchmarks like HotpotQA, MuSiQue, and 2WikiMultiHopQA, and supports a 32768 token context length.
Loading preview...
Model Overview
The metacognitive-behavioral-tuning/Qwen3-0.6B-gpt-oss-distill is a 0.8 billion parameter language model built upon the Qwen/Qwen3-0.6B base architecture. Its primary distinction lies in its training methodology: it undergoes a "gpt-oss-distill" process, which involves naive distillation of raw traces from a larger teacher model (gpt-oss-120b) through Supervised Fine-Tuning (SFT) followed by Generative Reinforcement Learning with Policy Optimization (GRPO).
Key Capabilities
- Multi-Hop Question Answering: Specifically tuned and optimized for complex question answering tasks that require synthesizing information from multiple sources or steps.
- Distilled Performance: Leverages knowledge from a larger, more capable teacher model, aiming to achieve strong performance in a smaller, more efficient package.
- Benchmark Performance: Evaluated on multi-hop QA datasets such as HotpotQA (in-domain) and MuSiQue / 2WikiMultiHopQA (out-of-domain).
Training Details
The model's training involved an SFT phase with a learning rate of 1e-4 and a batch size of 128, primarily using the HotpotQA dataset. This was subsequently followed by a GRPO phase to further refine its capabilities.
Good For
- Applications requiring efficient and accurate multi-hop question answering.
- Research into model distillation techniques for complex reasoning tasks.
- Deployment scenarios where a smaller model size is critical, but strong QA performance is still needed.