MAmmoTH2-7B: Enhanced Reasoning through Web-Scale Instruction Tuning

MAmmoTH2-7B, developed by TIGER-Lab, is a 7 billion parameter language model built on the Mistral architecture. It introduces an innovative instruction tuning approach that significantly boosts reasoning capabilities, especially in mathematical domains. The model achieves this by efficiently extracting and utilizing 10 million instruction-response pairs from a pre-training web corpus, a cost-effective method for acquiring high-quality instruction data.

Key Capabilities & Performance

• Enhanced Reasoning: MAmmoTH2-7B demonstrates substantial improvements in reasoning benchmarks. For instance, its performance on MATH tasks increased from 11% to 36.7% and on GSM8K from 36% to 68.4% compared to its base model, all without relying on domain-specific training data.
• Instruction Tuning: The model is fine-tuned using the WEBINSTRUCT dataset, focusing on improving its ability to follow complex instructions and generate accurate responses.
• Benchmark Results: Achieves notable scores across various reasoning and math-focused evaluations, including 29.0 on TheoremQA, 36.7 on MATH, 68.4 on GSM8K, and an average of 52.7 across multiple benchmarks.

Use Cases

MAmmoTH2-7B is particularly well-suited for applications requiring strong reasoning and problem-solving abilities, especially in mathematical contexts. Its generalist approach to math makes it a valuable tool for tasks ranging from open-ended math problems to multiple-choice questions. For more advanced capabilities, the MAmmoTH2-Plus variants, trained on additional public instruction datasets, offer even higher performance on reasoning and chatbot benchmarks.