Falcon12136/Ouro-1.4B

Ouro-1.4B: A Parameter-Efficient Looped Language Model

Ouro-1.4B is a 1.4 billion parameter Looped Language Model (LoopLM) developed by ByteDance, distinguished by its exceptional parameter efficiency. It is designed to achieve performance levels typically seen in 3-4 billion parameter standard transformers through an innovative architecture that employs iterative shared-weight computation.

Key Capabilities & Features

• Iterative Latent Reasoning: The model performs reasoning by recurrently processing information within its latent space, enhancing its analytical capabilities without a proportional increase in parameters.
• Adaptive Computation: Ouro-1.4B supports early exit mechanisms, allowing for dynamic allocation of computational resources based on the complexity of the task. This can be configured via total_ut_steps and early_exit_threshold in its config.json.
• Efficient Architecture: Based on a decoder-only Transformer, it features 24 layers, 4 recurrent steps, a hidden size of 2048, and a vocabulary of 49,152. It was trained on 7.7 trillion tokens, including web data, code, mathematics, and long-context documents.

Use Cases & Considerations

This model is primarily intended for research purposes, particularly in exploring efficient language processing and advanced reasoning techniques. Its ability to match larger models with fewer parameters makes it valuable for environments where computational resources are a constraint. Developers can adjust recurrent steps to balance performance and computation time. However, it's important to note that the adaptive exit feature is not currently supported by vLLM due to its inference optimizations. The model's training included a focus on long-context documents, with a context length extendable to 64K tokens.