Model Overview

The mlfoundations-dev/math-stratos-verified-scaled-0.25 is a 7.6 billion parameter language model, fine-tuned from the Qwen/Qwen2.5-7B-Instruct architecture. It features a significant context length of 131072 tokens, enabling it to process extensive inputs and generate detailed responses.

Key Characteristics

• Base Model: Fine-tuned from Qwen/Qwen2.5-7B-Instruct.
• Parameter Count: 7.6 billion parameters.
• Context Length: Supports a large context window of 131072 tokens.
• Specialized Training: Fine-tuned on the mlfoundations-dev/math-stratos-verified-scaled-0.25 dataset, suggesting an optimization for mathematical reasoning and verification tasks.

Training Details

The model was trained using specific hyperparameters:

• Learning Rate: 1e-05
• Optimizer: ADAMW_TORCH with betas=(0.9, 0.999) and epsilon=1e-08.
• Batch Sizes: A total train batch size of 96 (train_batch_size: 1, gradient_accumulation_steps: 3, num_devices: 32) and a total eval batch size of 256 (eval_batch_size: 8, num_devices: 32).
• Epochs: Trained for 3.0 epochs.
• Scheduler: Cosine learning rate scheduler with a warmup ratio of 0.1.

Potential Use Cases

Given its specialized training, this model is likely suitable for applications requiring:

• Solving complex mathematical problems.
• Verifying mathematical proofs or calculations.
• Assisting in educational tools for mathematics.
• Research in automated theorem proving or symbolic AI.