yingfanbot/gsm-lotus-llama3b
The yingfanbot/gsm-lotus-llama3b is a 3.2 billion parameter LOTUS model, fine-tuned from meta-llama/Llama-3.2-3B-Instruct. Developed by Ying Fan, Anej Svete, and Kangwook Lee, this model is specifically designed for enhanced reasoning, achieving 70.05% accuracy on the GSM8k-Aug mathematical reasoning benchmark. It utilizes a looped transformer architecture with direct LM-head supervision to bridge latent and explicit reasoning.
Loading preview...
Model Overview
The yingfanbot/gsm-lotus-llama3b is a 3.2 billion parameter LOTUS (Looped Transformers) model, fine-tuned from meta-llama/Llama-3.2-3B-Instruct. This model implements a novel architecture that bridges the gap between latent and explicit reasoning through direct LM-head supervision, as detailed in the paper "Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers" by Fan, Svete, and Lee.
Key Capabilities
- Enhanced Reasoning: Specifically designed to improve reasoning capabilities, particularly in mathematical problem-solving.
- GSM8k Performance: Achieves a 70.05% test accuracy on the GSM8k-Aug dataset, demonstrating strong performance in arithmetic reasoning.
- Looped Transformer Architecture: Incorporates a unique looped architecture with configurable latent blocks (
K=6), thought tokens per block (c_thought=25), and looped iterations (n_looped_iters=6).
Good For
- Mathematical Reasoning Tasks: Ideal for applications requiring robust mathematical problem-solving and step-by-step reasoning.
- Research in Reasoning Architectures: Useful for researchers exploring advanced transformer architectures for improved reasoning and interpretability.
- Benchmarking: Can be used as a baseline or comparison model for evaluating new reasoning-focused language models.