akumaburn/Open_Orca_Llama-3-8B-1K
akumaburn/Open_Orca_Llama-3-8B-1K is an 8 billion parameter Llama 3 model fine-tuned by akumaburn using the OpenOrca dataset. This model leverages Unsloth for accelerated training and features a 8192 token context window. It is optimized for general language understanding and generation tasks, following the Alpaca prompt format.
Loading preview...
Model Overview
akumaburn/Open_Orca_Llama-3-8B-1K is an 8 billion parameter language model developed by akumaburn, fine-tuned from unsloth/llama-3-8b-bnb-4bit. The training utilized the extensive OpenOrca dataset over 1000 steps with a batch size of 2 and 4 gradient accumulation steps. A key aspect of its development is the use of Unsloth and Huggingface's TRL library, which enabled a 2x faster training process.
Key Capabilities
- Context Window: Supports an 8192 token context size, allowing for processing longer inputs and generating more coherent, extended responses.
- Prompt Format: Adheres to the Alpaca prompt format, ensuring compatibility with common instruction-following paradigms.
- Quantized Versions: Includes GGUF quantizations for efficient deployment, with specific Q8_0 versions available for testing.
Performance Insights
While specific benchmarks for Open_Orca_Llama-3-8B-unsloth.Q8_0.gguf show MMLU-Test at 39.3818 and Arc-Challenge at 42.1405, it's notable that the base llama-3-8b-bnb-4bit.Q8_0.gguf and Meta-Llama-3-8B.Q8_0.gguf demonstrate slightly higher performance in some metrics like MMLU and Arc-Easy, suggesting the fine-tuning focuses on specific instruction-following capabilities derived from the OpenOrca dataset rather than raw benchmark uplift across all categories. The model is licensed under Apache-2.0.
Top 3 parameter combinations used by Featherless users for this model. Click a tab to see each config.