MAmmoTH2-8B-Plus: Enhanced Reasoning through Web-Scale Instruction Tuning

MAmmoTH2-8B-Plus, developed by TIGER-Lab, is an 8 billion parameter language model built on the Llama-3 architecture. It is a refined version of the MAmmoTH2 series, which focuses on dramatically improving the reasoning capabilities of LLMs through an innovative instruction tuning approach. The core innovation lies in efficiently harvesting 10 million high-quality instruction-response pairs from the pre-training web corpus, a cost-effective method for acquiring large-scale instruction data.

Key Capabilities & Differentiators

• Superior Reasoning: MAmmoTH2 models show significant performance boosts on reasoning benchmarks. For instance, the base MAmmoTH2-7B (Mistral) improved from 11% to 36.7% on MATH and 36% to 68.4% on GSM8K without domain-specific training.
• MAmmoTH2-Plus Enhancement: The "Plus" variants, including MAmmoTH2-8B-Plus, undergo further training on public instruction tuning datasets, leading to even higher performance across reasoning and chatbot benchmarks.
• Cost-Effective Data Acquisition: The project introduces a novel and efficient method for gathering large-scale, high-quality instruction data directly from the web, offering a fresh perspective on enhancing LLM reasoning.
• Strong Benchmark Performance: MAmmoTH2-8B-Plus achieves notable scores on various evaluation datasets, including 43.0 on MATH, 85.2 on GSM8K, and 59.4 average across multiple reasoning benchmarks.

Good For

• Applications requiring strong mathematical and general reasoning.
• Developers seeking models with enhanced instruction-following capabilities derived from web-scale data.
• Use cases where a balance between model size and advanced reasoning performance is crucial.