Overview

The mlfoundations-dev/deepmath model is a 7.6 billion parameter language model, fine-tuned from the Qwen/Qwen2.5-7B-Instruct base model. It has been specifically adapted using the mlfoundations-dev/deepmath dataset, indicating a specialization in areas related to deep learning foundations and mathematics.

Key Characteristics

• Base Model: Qwen/Qwen2.5-7B-Instruct
• Parameter Count: 7.6 billion parameters
• Context Length: 131072 tokens, allowing for extensive input processing.
• Training Data: Fine-tuned on the mlfoundations-dev/deepmath dataset.

Training Details

The model was trained with a learning rate of 8e-05, a total batch size of 512, and utilized 128 devices. The training procedure involved 5 epochs, using the AdamW optimizer with a cosine learning rate scheduler and a warmup ratio of 0.1. This configuration suggests a robust training regimen aimed at enhancing its specialized capabilities.

Intended Use Cases

Given its fine-tuning on a deepmath dataset, this model is likely optimized for tasks involving:

• Mathematical problem-solving and reasoning.
• Understanding and generating technical content related to deep learning and mathematics.
• Applications requiring a deep comprehension of scientific and theoretical concepts.