Model Overview

The stratos_unverified_mix_2nodes model is a 7.6 billion parameter language model developed by mlfoundations-dev. It is a fine-tuned variant of the robust Qwen/Qwen2.5-7B-Instruct architecture, designed to leverage its strong base capabilities. This model is distinguished by its training on the mlfoundations-dev/stratos_unverified_mix dataset, suggesting a focus on specific data characteristics or tasks represented within that dataset.

Key Characteristics

• Base Model: Fine-tuned from Qwen/Qwen2.5-7B-Instruct.
• Parameter Count: 7.6 billion parameters.
• Context Length: Supports a substantial context window of 131,072 tokens.
• Training Data: Specialized training on the mlfoundations-dev/stratos_unverified_mix dataset.

Training Details

The model was trained using a learning rate of 1e-05, a total batch size of 96 (with 16 devices and 6 gradient accumulation steps), and a cosine learning rate scheduler with a 0.1 warmup ratio over 3 epochs. The training utilized Transformers 4.46.1, Pytorch 2.3.0, Datasets 3.1.0, and Tokenizers 0.20.3.