DuoNeural/Qwen2.5-Coder-3B-SFT-WebCode
DuoNeural/Qwen2.5-Coder-3B-SFT-WebCode is a 3.1 billion parameter SFT fine-tune by DuoNeural, based on the Qwen2.5-Coder-3B-Instruct model. This model is specifically fine-tuned using the DuoNeural/Gemma4-E2B-SFT-WebCode dataset, focusing on code-related tasks. It is designed for exploring emergent behaviors in small language models, particularly in code generation and understanding contexts.
Loading preview...
Overview
DuoNeural/Qwen2.5-Coder-3B-SFT-WebCode is a 3.1 billion parameter language model, fine-tuned by DuoNeural. It is built upon the Qwen/Qwen2.5-Coder-3B-Instruct base model and has undergone Supervised Fine-Tuning (SFT) using the specialized DuoNeural/Gemma4-E2B-SFT-WebCode dataset. This model is part of DuoNeural's research into emergent behaviors within small language models.
Training Details
The model was trained using LoRA with a rank of 16 and an alpha of 32, over 3 epochs, with a learning rate of 2e-4 and an effective batch size of 16.
Benchmark Results
Evaluation was conducted using lm_eval 0.4.x on GSM8K and ARC-Challenge datasets. While the model shows a decrease in GSM8K performance compared to its baseline, its ARC-Challenge scores remain consistent.
| Model | GSM8K flex | ARC-norm | ARC-acc |
|---|---|---|---|
| Baseline | 0.5807 | 0.4957 | 0.4590 |
| Qwen2.5-Coder-3B-SFT-WebCode | 0.3207 | 0.4957 | 0.4590 |
Use Cases
This model is suitable for developers and researchers interested in:
- Code-related tasks: Leveraging its SFT on a web code dataset.
- Exploring small language model capabilities: Particularly in the context of code generation and understanding.
- Further fine-tuning: As a base for more specialized code-centric applications.