SubMaroon/Orion-Kitchoon-26B-A4B
Orion-Kitchoon-26B-A4B is a 26 billion parameter causal language model developed by SubMaroon, based on TheDrummer's Orion 26B A4B v1.1 with a light merge of Kitchoon's OV LoRA. This model maintains the full strength of the base Orion model, incorporating Kitchoon's OV LoRA at layers 10-29 with a weight of 0.32. It is designed for general language understanding and generation tasks, though the merge was noted as largely unsuccessful in improving performance.
Loading preview...
Orion-Kitchoon-26B-A4B Overview
Orion-Kitchoon-26B-A4B is a 26 billion parameter language model developed by SubMaroon, built upon TheDrummer's Orion 26B A4B v1.1 base model. It incorporates a light merge of Kitchoon's OV LoRA to potentially enhance its capabilities, specifically by adding Kitchoon's OV LoRA to layers 10-29 at a weight of 0.32. The model retains the full strength of the original Orion base.
Key Characteristics
- Base Model: TheDrummer's Orion 26B A4B v1.1.
- Parameter Count: 26 billion parameters.
- Context Length: Supports a context length of 32768 tokens.
- Merge Strategy: Integrates Kitchoon's OV LoRA (layers 10-29) at a 0.32 weight.
Considerations for Use
While the merge aimed to combine strengths, the developer notes that the integration of Kitchoon's OV LoRA was largely unsuccessful in yielding significant improvements. Users should evaluate its performance for their specific applications, keeping in mind that the primary strength remains that of the underlying Orion 26B A4B v1.1 model.