Sorihon/Amended-Journey-24B
Sorihon/Amended-Journey-24B is a 24 billion parameter language model created by Sorihon, merged using the Karcher Mean method with TheDrummer/Cydonia-24B-v4.3 as its base. This model integrates components from two distinct local models, /home/sorihon/Documents/CYDR-V2/ and /home/sorihon/Documents/CYDR-V3/, to potentially enhance its capabilities. With a context length of 32768 tokens, it is designed for general language tasks, leveraging its merged architecture for improved performance.
Loading preview...
Overview
Sorihon/Amended-Journey-24B is a 24 billion parameter language model developed by Sorihon. It was constructed using the Karcher Mean merge method, a technique aimed at combining the strengths of multiple pre-trained models. The base model for this merge was TheDrummer/Cydonia-24B-v4.3.
Merge Details
This model's architecture is a result of merging two distinct local models, specifically /home/sorihon/Documents/CYDR-V2/ and /home/sorihon/Documents/CYDR-V3/. The merge process utilized mergekit with a configuration that included:
- Merge Method: Karcher Mean
- Base Model: TheDrummer/Cydonia-24B-v4.3
- Tokenizer: Union of source tokenizers
- Normalization: Enabled
- Quantization:
int8_maskapplied - Data Type:
bfloat16
Key Characteristics
- Parameter Count: 24 billion parameters.
- Context Length: Supports a substantial context window of 32768 tokens.
- Development Focus: Represents an attempt to create a robust merge by carefully combining model components rather than relying on blind guessing, aiming for improved performance and coherence.
Potential Use Cases
Given its merged nature and substantial parameter count, Amended-Journey-24B is likely suitable for a broad range of natural language processing tasks, including text generation, summarization, and question answering, where the combined knowledge of its constituent models could offer advantages.