OpenLLaMA 7B Scaled: An Open Reproduction of LLaMA
emozilla/open_llama_7b-scaled is a 7 billion parameter language model from OpenLM Research, designed as an open-source reproduction of Meta AI's LLaMA. A key feature of this model is its integration of Scaled Rotary Embeddings, which enables flexible context window configuration. While the default context length is 2048 tokens, users can easily extend this to larger values like 4096 or 8192 tokens by adjusting the max_position_embeddings in the model's configuration.
Key Capabilities & Features
- LLaMA Architecture Reproduction: Faithfully reproduces the LLaMA model architecture and training hyperparameters.
- Extended Context Window: Utilizes Scaled Rotary Embeddings for configurable context lengths, offering greater flexibility for longer sequences.
- Extensive Training Data: Trained on 1 trillion tokens from the RedPajama dataset, an open reproduction of the LLaMA training data.
- Competitive Performance: Achieves performance comparable to the original LLaMA 7B and GPT-J 6B models across various benchmarks, and in some cases, outperforms them.
- Permissive Licensing: Released under the Apache 2.0 license, allowing for broad usage and integration.
When to Use This Model
- General Language Tasks: Suitable for a wide range of natural language generation and understanding applications.
- Research and Development: Ideal for researchers and developers looking for an open-source, permissively licensed LLaMA-like model.
- Applications Requiring Longer Context: Beneficial for use cases where processing and generating longer text sequences are crucial, thanks to its configurable context window.
- Benchmarking: Can be used as a strong baseline for evaluating new models or techniques, offering a robust comparison point against established models like LLaMA and GPT-J.