Overview

mlfoundations-dev/stratos-unverified-mix-scaled-0.25 is a 7.6 billion parameter language model, fine-tuned from the established Qwen/Qwen2.5-7B-Instruct architecture. This model distinguishes itself through its specific training on the mlfoundations-dev/stratos-unverified-mix-scaled-0.25 dataset, implying a tailored performance profile for tasks related to this data mixture. It supports a substantial context window of 131,072 tokens, enabling it to handle and generate extensive textual content while maintaining coherence.

Training Details

The model was trained using a learning rate of 1e-05, a total batch size of 96 (with 3 gradient accumulation steps across 32 devices), and 3 epochs. The optimizer used was ADAMW_TORCH with standard betas and epsilon, and a cosine learning rate scheduler with a 0.1 warmup ratio. This configuration suggests a robust training regimen aimed at optimizing performance on its specific fine-tuning dataset.

Key Characteristics

• Base Model: Qwen/Qwen2.5-7B-Instruct
• Parameter Count: 7.6 billion
• Context Length: 131,072 tokens
• Fine-tuning Dataset: mlfoundations-dev/stratos-unverified-mix-scaled-0.25

Intended Use

Given its fine-tuning on a specific dataset and large context window, this model is likely best suited for applications that benefit from processing extensive inputs and require outputs aligned with the characteristics of the stratos-unverified-mix-scaled-0.25 data. Developers should consider its specialized training for tasks where this particular data mixture provides an advantage.