Overview
The hyunseoki/verl-math-transfer-7bi-to-3bi-fix03 model is a 7.6 billion parameter language model built on the Qwen2ForCausalLM architecture. It is the result of a math transfer experiment conducted using the verl framework, specifically designed to explore the transfer of mathematical capabilities from a larger 7 billion parameter model to a smaller 3 billion parameter configuration.
This repository contains various exported Hugging Face checkpoints for the 7B-to-3B fix_0_3 configuration, with the main branch currently pointing to the step-130 checkpoint. Users can load specific checkpoint revisions, such as step-010 through step-130, to analyze the progression of the transfer learning process.
Key Characteristics
- Architecture: Qwen2ForCausalLM.
- Parameter Count: 7.6 billion parameters.
- Purpose: Focused on mathematical transfer learning experiments.
- Checkpoints: Provides multiple step revisions, allowing for granular analysis of training progress.
- Export Format: Checkpoints are exported from
verl FSDP shards into Hugging Face safetensors format.
Intended Use Cases
This model is particularly suited for:
- Research in mathematical reasoning: Investigating how mathematical knowledge transfers between models of different sizes.
- Experimentation with model scaling: Studying the effects of down-scaling models while retaining specific capabilities.
- Development of math-focused LLMs: As a base or reference for further fine-tuning on mathematical tasks.