cstr/llama3-discolm-orca
The cstr/llama3-discolm-orca is an 8 billion parameter language model, created by cstr, formed by merging three Llama 3-based models: Llama-3-Orca-1.0-8B, llama-3-neural-chat-v1-8b, and Llama3_DiscoLM_German_8b_v0.1_experimental. This model merge, with an 8192 token context length, was developed as a proof-of-concept to combine the strengths of its constituent models. It is suitable for general language tasks, potentially with enhanced German language capabilities due to one of its merged components.
Loading preview...
Overview
cstr/llama3-discolm-orca is an 8 billion parameter language model created by cstr as a proof-of-concept merge. It combines three distinct Llama 3-based models: Locutusque/Llama-3-Orca-1.0-8B, Locutusque/llama-3-neural-chat-v1-8b, and DiscoResearch/Llama3_DiscoLM_German_8b_v0.1_experimental. The merge was performed using the dare_ties method with specific density and weight parameters for each constituent model, as detailed in the provided LazyMergekit configuration.
Key Characteristics
- Architecture: Based on the Llama 3 family, inheriting its foundational capabilities.
- Parameter Count: 8 billion parameters.
- Context Length: Supports an 8192 token context window.
- Merge Method: Utilizes the
dare_tiesmerge method to combine the weights of its components. - Multilingual Potential: Includes a German-focused Llama 3 model, suggesting potential for improved performance in German language tasks.
Usage and Licensing
This model can be used with the Hugging Face transformers library for text generation tasks. The model's license is the Llama 3 Community Licence, which propagates from its constituent Llama 3 derivatives. As a model merge, cstr is considered the provider under the EU AI Act, with obligations related to copyright policy and training content, though no new training data was used in the merging process.