KristianS7/Ouro-2.6B

Ouro-2.6B: A Parameter-Efficient Looped Language Model

Ouro-2.6B is a 2.6 billion parameter Looped Language Model (LoopLM) developed by ByteDance, distinguishing itself through its unique architecture that emphasizes parameter efficiency. It is designed to match the performance of larger 3-4 billion parameter standard transformers by employing iterative shared-weight computation.

Key Capabilities and Features

• Exceptional Parameter Efficiency: Achieves performance equivalent to larger models (3-4B parameters) with a smaller 2.6B parameter count.
• Iterative Latent Reasoning: Utilizes recurrent computation in its latent space to perform reasoning tasks.
• Adaptive Computation: Incorporates early exit mechanisms, allowing for dynamic allocation of computational resources based on task complexity.
• Configurable Recurrent Steps: Users can adjust total_ut_steps (default 4) to balance performance and computation time.
• Adaptive Exit Threshold: The early_exit_threshold (default 1.0) controls how aggressively the model exits early, with lower values encouraging earlier termination.

Architecture and Training

Built on a decoder-only Transformer architecture, Ouro-2.6B features 24 layers, a hidden size of 2048, and Multi-Head Attention. It was trained on 7.7 trillion tokens, including web data, code, mathematics, and long-context documents, across multiple stages. The model supports a context length of 4K during training, extendable to 64K.

Important Considerations

• Research Use Only: This model is intended solely for research purposes.
• Transformers Compatibility: Requires transformers==5.9.0 for full compatibility.
• vLLM Limitation: The adaptive exit feature is not currently supported when using vLLM; the model will execute all recurrent steps.