tokyotech-llm/Llama-3.1-8B-math-ablation-exp2-LR2.5e-5-WD0.1-iter0002500

Model Overview

This model is a continual pre-training of Llama-3.1-8B by tokyotech-llm, specifically developed as part of the SwallowMath ablation experiments (experiment 2) to evaluate mathematical reasoning and problem-solving. It was trained on a total of 50 billion tokens.

Training Details

The training dataset comprised a unique mix:

• Mathematical Data: ~4.8% from SwallowMath (finemath-4+ rewritten)
• Code Data: ~13.1% from SwallowCode
• Multilingual Text: ~82% including Japanese and English corpora

The model utilizes the Llama-3 tokenizer and was trained with bfloat16 precision on 64 NVIDIA H100 GPUs using Megatron-LM.

Evaluation and Performance

Evaluated using the setup described in the SwallowMath paper, this model's performance was tracked across various benchmarks at different token counts (10B, 20B, 30B, 40B, 50B). Key benchmarks include:

• Mathematical Reasoning: GSM8K, MATH
• Code Generation: HumanEval
• General Tasks: MMLU, BBH, OpenBookQA, TriviaQA, HellaSwag, SQuAD 2.0, XWINO

Results show a progressive improvement in mathematical and general reasoning tasks as training tokens increased, with the final checkpoint at 50B tokens achieving 0.6535 on GSM8K and 0.3160 on MATH.

Use Cases

This model is particularly well-suited for:

• Mathematical Problem Solving: Excelling in tasks requiring quantitative reasoning.
• Code Generation: Benefiting from its significant code data exposure.
• Research in LLM Training: Ideal for researchers studying the impact of data mix on mathematical and reasoning capabilities.