lino-levan/qwen3-1.7b-smoltalk
The lino-levan/qwen3-1.7b-smoltalk model is a 2 billion parameter Qwen3-based causal language model, fine-tuned by lino-levan on the HuggingFaceTB/smoltalk dataset. With a 40960 token context length, this model is specifically optimized for tasks related to the smoltalk dataset's characteristics. It aims to provide focused performance within its specialized training domain.
Loading preview...
Overview
lino-levan/qwen3-1.7b-smoltalk is a 2 billion parameter language model built on the Qwen3 architecture. It has been specifically fine-tuned by lino-levan using the HuggingFaceTB/smoltalk dataset. The training involved 2 epochs with an effective batch size of 128, utilizing 4x H100 GPUs.
Key Characteristics
- Architecture: Qwen3 base model.
- Parameters: Approximately 2 billion.
- Context Length: Supports a substantial context window of 40960 tokens.
- Training Data: Fine-tuned exclusively on the smoltalk dataset.
- Training Details: Achieved a final training loss of 0.6432 over 3,300 steps.
Performance Benchmarks
While specialized, the model's current benchmark scores indicate areas for further development:
- AIME25: 0%
- GPQA: 24.8%
- GSM8K: 54.2%
- IFBench: 18.3%
- IFEval: 55%
- MMLU-Pro: 22.8%
- Multi-IF: 32.5%
Use Cases
This model is primarily suitable for:
- Research and experimentation with models fine-tuned on specific, smaller datasets like smoltalk.
- Exploring the impact of targeted fine-tuning on a Qwen3 base model.
- Developing applications that align closely with the smoltalk dataset's content and style.