OpenLLaMA 7B v2: An Open Reproduction of LLaMA

OpenLLaMA 7B v2 is a 7 billion parameter causal language model developed by openlm-research, serving as an open-source, permissively licensed reproduction of Meta AI's LLaMA. This model is trained on 1 trillion tokens, utilizing a diverse data mixture including the Falcon refined-web dataset, StarCoder dataset, and select components of the RedPajama dataset (Wikipedia, ArXiv, Books, StackExchange).

Key Capabilities & Features

• LLaMA Compatibility: Designed as a drop-in replacement for LLaMA in existing implementations.
• Permissive Licensing: Released under the Apache 2.0 license for both model weights and the EasyLM training framework.
• Training Methodology: Follows the exact preprocessing steps and training hyperparameters of the original LLaMA paper, with the primary difference being the use of open datasets.
• Performance: Exhibits comparable performance to the original LLaMA 7B and GPT-J 6B across a majority of evaluated tasks, and outperforms them in some instances, as shown in LM-Eval-Harness benchmarks.

Good For

• Developers seeking a LLaMA-like model with a permissive license.
• Research and experimentation with large language models, particularly those interested in open reproductions.
• Applications requiring a 7B parameter model with strong general language understanding and generation capabilities.