anonymous-bird-72/Ouro-1.4B
Ouro-1.4B is a 1.4 billion parameter Looped Language Model (LoopLM) developed by ByteDance, designed for exceptional parameter efficiency. It achieves performance comparable to standard 3-4B parameter transformers by utilizing iterative shared-weight computation and recurrent steps in its latent space. This model performs reasoning through recurrent computation and supports adaptive computation with early exit mechanisms. It is intended for research purposes, particularly in exploring efficient language model architectures and latent reasoning.
Loading preview...
Ouro-1.4B: A Looped Language Model for Efficient Reasoning
Ouro-1.4B, developed by ByteDance, is a 1.4 billion parameter Looped Language Model (LoopLM) that introduces a novel approach to achieving high performance with significantly fewer parameters. It leverages iterative shared-weight computation to perform recurrent steps in its latent space, enabling efficient reasoning.
Key Capabilities
- Exceptional Parameter Efficiency: Matches the performance of standard 3-4 billion parameter transformers using only 1.4 billion parameters.
- Iterative Latent Reasoning: Employs recurrent computation within its latent space to perform reasoning tasks.
- Adaptive Computation: Supports configurable recurrent steps (
total_ut_steps) and anearly_exit_thresholdfor dynamic compute allocation, allowing a trade-off between performance and computation time. - Robust Training: Trained on a diverse 7.7 trillion token dataset, including web data, code, mathematics, and long-context documents.
Good For
- Research into Efficient LLMs: Ideal for exploring parameter-efficient architectures and recurrent computation in language models.
- Applications requiring reduced computational overhead: When performance similar to larger models is needed with a smaller footprint.
- Experimentation with adaptive inference: Utilizing the configurable recurrent steps and early exit mechanisms to optimize inference.
This model is primarily intended for research purposes, as detailed in its Paper and Project Page. Users should note that transformers<4.56.0 is recommended for compatibility, and vLLM currently does not support the adaptive exit feature.