Matt300209/Ouro-modified-config

Ouro-1.4B: A Parameter-Efficient Looped Language Model

Ouro-1.4B, developed by ByteDance, is a 1.4 billion parameter Looped Language Model (LoopLM) designed for exceptional parameter efficiency. It demonstrates performance comparable to larger 3-4 billion parameter standard transformers by employing iterative shared-weight computation and recurrent steps for latent reasoning. The model is built on a decoder-only Transformer architecture with 24 layers, a hidden size of 2048, and a vocabulary of 49,152, supporting a context length of 4K (extendable to 64K).

Key Capabilities & Features

• Exceptional Parameter Efficiency: Achieves performance similar to larger models with significantly fewer parameters.
• Iterative Latent Reasoning: Performs reasoning through recurrent computation in its latent space.
• Adaptive Computation: Features configurable recurrent steps (total_ut_steps) and an early_exit_threshold to dynamically manage computation, allowing a trade-off between performance and speed.
• Configurable Behavior: Users can adjust recurrent steps via the config.json to optimize for specific needs.
• Extensive Training: Trained on 7.7 trillion tokens across multiple stages, including web data, code, mathematics, and long-context documents.

Use Cases & Considerations

• Research Purposes: Primarily intended for research into parameter-efficient language models and recurrent computation.
• Resource-Constrained Environments: Its efficiency makes it suitable for scenarios where computational resources are limited but strong performance is still required.
• Exploration of Adaptive Inference: Ideal for experimenting with dynamic compute allocation and early exit strategies in LLMs.

Note: The adaptive exit feature is not currently supported by vLLM, meaning the model will execute all recurrent steps when used with vLLM.