yingfanbot/gsm-lotus-llama3b-nl
The yingfanbot/gsm-lotus-llama3b-nl is a 3.2 billion parameter language model, fine-tuned from Meta's Llama-3.2-3B-Instruct. Developed by Ying Fan, Anej Svete, and Kangwook Lee, this model utilizes the LOTUS architecture, specifically trained with direct step-aligned supervision on natural-language chain-of-thought. It excels at mathematical reasoning tasks, achieving 68.54% accuracy on the GSM8K benchmark by supervising latent blocks against natural-language reasoning steps.
Loading preview...
Model Overview
The yingfanbot/gsm-lotus-llama3b-nl is a 3.2 billion parameter language model based on the meta-llama/Llama-3.2-3B-Instruct architecture. It incorporates the LOTUS (Looped Transformers) framework, which is designed to bridge the gap between latent and explicit reasoning. Unlike other LOTUS implementations, this model is uniquely fine-tuned using natural-language chain-of-thought supervision, directly aligning latent blocks with human-readable reasoning steps rather than formatted mathematical expressions.
Key Capabilities & Features
- Enhanced Reasoning: Specialized in mathematical problem-solving, achieving 68.54% accuracy on the GSM8K (GSM8k-Aug) test set.
- LOTUS Architecture: Utilizes a latent configuration with K=6 blocks and a
c_thoughtof 50 tokens per block, optimized for natural-language reasoning steps. - Direct Supervision: Employs direct step-aligned supervision through the base LM head, avoiding auxiliary decoders or CODI methods.
When to Use This Model
This model is particularly well-suited for applications requiring robust natural-language mathematical reasoning. Its training methodology, focusing on natural-language chain-of-thought, makes it a strong candidate for tasks where understanding and generating step-by-step explanations for problem-solving are crucial. Developers can leverage its specialized reasoning capabilities for educational tools, automated problem solvers, or any system benefiting from explicit, natural-language reasoning traces.