daksh-neo/qwen-to-gemma-math
The daksh-neo/qwen-to-gemma-math model, developed by NEO, is a 5.1 billion parameter Gemma 4-based language model with a 32768 token context length. It is specifically optimized for mathematical reasoning tasks, achieving 75.0% accuracy on the GSM8K benchmark. This model was created through knowledge distillation from Qwen3-plus and LoRA fine-tuning on the full GSM8K training set, outperforming its Gemma baseline by 4.0 percentage points.
Loading preview...
Model Overview
This model, daksh-neo/qwen-to-gemma-math, is a 5.1 billion parameter Gemma 4-based language model developed autonomously by NEO. It specializes in mathematical reasoning, particularly for grade school math problems, and was created using a unique knowledge distillation and fine-tuning pipeline.
Key Capabilities & Performance
- Enhanced Mathematical Reasoning: Achieves 75.0% accuracy on the GSM8K benchmark, outperforming the
google/gemma-4-E2B-itbaseline by +4.0 percentage points. - Knowledge Distillation: Distills complex mathematical chain-of-thought reasoning from the more powerful Qwen3-plus (teacher model) into the smaller Gemma 4 2B (student model).
- Autonomous Development: The entire pipeline, including design, trace generation, fine-tuning, and evaluation, was executed autonomously by NEO.
- LoRA Fine-Tuning: Utilizes LoRA (r=16, alpha=32) on the full GSM8K training dataset (7,473 samples) to efficiently transfer knowledge.
Ideal Use Cases
- Mathematical Problem Solving: Excellent for applications requiring step-by-step arithmetic and logical reasoning.
- Educational Tools: Can be integrated into systems for teaching or assisting with grade school mathematics.
- Resource-Efficient Math AI: Provides strong mathematical capabilities in a more compact model size compared to larger teacher models, suitable for deployment where computational resources are a consideration.