OpenLLaMA 13B: An Open Reproduction of LLaMA

This model, sharpbai/open_llama_13b, is a 13 billion parameter large language model developed by OpenLM Research. It is an open-source, permissively licensed reproduction of Meta AI's LLaMA architecture, trained on 1 trillion tokens from the RedPajama dataset. The training methodology closely follows the original LLaMA paper, including architecture, context length (4096 tokens), and hyperparameters, with the primary difference being the use of the RedPajama dataset.

Key Capabilities & Features

• LLaMA Architecture Reproduction: Faithfully replicates the LLaMA model architecture.
• Permissive Licensing: Released under the Apache 2.0 license, allowing for broad use.
• Extensive Training Data: Trained on 1 trillion tokens from the RedPajama dataset.
• Comparable Performance: Achieves performance comparable to the original LLaMA 13B and GPT-J 6B models across a range of evaluation tasks, including ARC Challenge, HellaSwag, and BoolQ.
• Hugging Face Transformers Integration: Easily loadable and usable with the Hugging Face transformers library, though it's advised to use LlamaTokenizer or use_fast=False for AutoTokenizer due to observed tokenization issues with the fast tokenizer.

When to Use This Model

• Open-source LLaMA Alternative: Ideal for developers seeking a LLaMA-like model with a permissive license.
• General Language Generation: Suitable for various natural language processing tasks where a 13B parameter model is appropriate.
• Research and Development: Provides a strong baseline for further research, fine-tuning, or experimentation with LLaMA-style models.