Overview

This model, mlfoundations-dev/teacher_code_qwq, is a 7.6 billion parameter language model fine-tuned from the Qwen/Qwen2.5-7B-Instruct base. It was developed by mlfoundations-dev with a focus on code-related applications.

Training Details

The model underwent fine-tuning using the mlfoundations-dev/teacher_code_qwq dataset. Key training hyperparameters included a learning rate of 4e-05, a total batch size of 128 (with 64 devices and 2 gradient accumulation steps), and 5 epochs. The optimizer used was AdamW with specific beta and epsilon values, and a cosine learning rate scheduler with a 0.1 warmup ratio.

Intended Use

While specific intended uses and limitations require more information, the fine-tuning on a code-centric dataset suggests its primary application is in code generation, comprehension, or related programming tasks. Developers seeking a specialized model for such applications, building upon the Qwen2.5-7B-Instruct architecture, may find this model suitable.