OpenOrca-Platypus2-13B: A Merged Llama 2 Model

OpenOrca-Platypus2-13B is a 13 billion parameter auto-regressive language model built upon the Llama 2 transformer architecture. It is a strategic merge of two distinct models: garage-bAInd/Platypus2-13B and Open-Orca/OpenOrcaxOpenChat-Preview2-13B. This combination aims to leverage the strengths of both, resulting in a model that often surpasses the individual components.

Key Capabilities & Performance

This model exhibits strong performance across various benchmarks, as evaluated using the Language Model Evaluation Harness:

• HuggingFace Leaderboard Average: Achieves an average score of 64.56, with notable results including 59.5 on MMLU (5-shot) and 62.88 on ARC (25-shot).
• AGIEval Performance: Demonstrates 112% of the base Preview2 model's performance, averaging 0.463. A significant improvement was observed in LSAT Logical Reasoning.
• BigBench-Hard Performance: Shows 105% of the base Preview2 model's performance, averaging 0.442.

Training & Licensing

The model was instruction fine-tuned using LoRA. The Platypus2-13B component was trained on STEM and logic-based datasets, while OpenOrcaxOpenChat-Preview2-13B utilized a refined subset of GPT-4 data from the OpenOrca dataset. The base weights for Platypus2-13B are under a Non-Commercial Creative Commons license (CC BY-NC-4.0), while OpenOrcaxOpenChat-Preview2-13B uses the Llama 2 Commercial license.

Good for

• Applications requiring strong logical reasoning and problem-solving.
• Tasks benefiting from improved performance on benchmarks like AGIEval and BigBench-Hard.
• Developers seeking a Llama 2-based model with enhanced general intelligence capabilities.