anonymous-bird-72/Ouro-2.6B
Ouro-2.6B by ByteDance is a 2.6 billion parameter Looped Language Model (LoopLM) designed for exceptional parameter efficiency. It achieves performance comparable to larger 3-4B standard transformers by utilizing iterative shared-weight computation and recurrent latent reasoning. This model supports adaptive computation with configurable recurrent steps and early exit mechanisms, making it suitable for applications requiring efficient reasoning and dynamic compute allocation.
Loading preview...
Ouro-2.6B: A Parameter-Efficient Looped Language Model
Ouro-2.6B, developed by ByteDance, is a 2.6 billion parameter Looped Language Model (LoopLM) that distinguishes itself through exceptional parameter efficiency. It is designed to match the performance of larger 3-4 billion parameter standard transformers by employing iterative shared-weight computation.
Key Capabilities & Features
- Iterative Latent Reasoning: The model performs reasoning through recurrent computation within its latent space, allowing for more efficient processing.
- Adaptive Computation: It supports configurable recurrent steps (
total_ut_steps) and an adaptive early exit mechanism (early_exit_threshold), enabling dynamic allocation of computational resources based on task complexity. - Efficient Architecture: Based on a decoder-only Transformer with 24 layers, 4 recurrent steps, and a 2048 hidden size, utilizing SwiGLU activation and RoPE for position embeddings.
- Extensible Context Length: While trained with a 4K context length, it is designed to be extendable up to 64K tokens.
Training & Configuration
Ouro-2.6B was trained on 7.7 trillion tokens across multiple stages, including pre-training, CT annealing, long-context training, and mid-training, using a diverse dataset comprising web data, code, mathematics, and long documents. Users can customize the model's behavior by adjusting total_ut_steps and early_exit_threshold in the config.json to balance performance and computation time. Note that vLLM currently bypasses the adaptive exit feature, always executing the full number of recurrent steps.
Use Cases
This model is particularly well-suited for research purposes where parameter efficiency and iterative reasoning are critical. Its adaptive computation features make it valuable for scenarios requiring dynamic resource management and efficient processing of complex tasks.