openlm-research/open_llama_7b_v2_step_460000

OpenLLaMA 7B v2: An Open Reproduction of LLaMA

OpenLLaMA 7B v2, developed by OpenLM Research, is a 7 billion parameter large language model designed as a permissively licensed, open-source reproduction of Meta AI's LLaMA. This model is part of a series that includes 3B, 7B, and 13B variants, offering a drop-in replacement for LLaMA in existing implementations.

Key Capabilities

• LLaMA Architecture Reproduction: Follows the exact model architecture, context length, training steps, learning rate schedule, and optimizer as the original LLaMA paper.
• Diverse Training Data: The v2 models are trained on a mixture of the Falcon refined-web dataset, StarCoder dataset, and selected parts of the RedPajama dataset (Wikipedia, ArXiv, Book, StackExchange), totaling 1 trillion tokens.
• Comparable Performance: Achieves performance comparable to the original LLaMA 7B and GPT-J 6B across a majority of evaluated tasks, and in some cases, outperforms them.
• Permissive Licensing: Both the training framework (EasyLM) and the model weights are released under the Apache 2.0 license, enabling broad usage.
• Hugging Face Transformers Integration: Weights are available in PyTorch format for direct use with the Hugging Face transformers library, with specific guidance provided for tokenizer usage.

Good For

• Researchers and Developers: Seeking an open-source, permissively licensed alternative to LLaMA for research and development.
• General NLP Tasks: Suitable for a wide array of language understanding and generation tasks due to its broad training data and LLaMA-like architecture.
• Benchmarking and Comparison: Useful for evaluating and comparing against other large language models, particularly LLaMA and GPT-J.
• EasyLM Framework Users: Integrates seamlessly with the EasyLM framework for training and fine-tuning large language models.