Sorihon/Celestial-Order-24B-2.75
Celestial-Order-24B-2.75 is a 24 billion parameter language model developed by Sorihon, created through a DARE TIES merge of pre-trained models, including Sorihon/CYDR-24B. This model is a refined iteration, building upon Celestial-Order-V2.5, and is designed for general language understanding and generation tasks with a context length of 32768 tokens.
Loading preview...
Model Overview
Celestial-Order-24B-2.75 is a 24 billion parameter language model developed by Sorihon. It is a merged model, specifically created using the DARE TIES merge method, which combines the strengths of multiple pre-trained language models. This version, V2.75, builds upon a previous iteration, Celestial-Order-V2.5, serving as its base.
Merge Details
The model was constructed by merging:
- Sorihon/CYDR-24B
The merge process utilized specific density and weight parameters for each component model, with a bfloat16 data type for efficiency. The DARE TIES method is known for its ability to effectively combine model weights, aiming to enhance overall performance and capabilities.
Key Characteristics
- Parameter Count: 24 billion parameters, offering a balance between performance and computational requirements.
- Context Length: Supports a substantial context window of 32768 tokens, enabling the processing and generation of longer texts while maintaining coherence.
- Development Method: Leverages
mergekitand the DARE TIES technique, indicating a focus on combining and refining existing model architectures rather than training from scratch.
Potential Use Cases
Given its architecture and parameter count, Celestial-Order-24B-2.75 is suitable for a variety of natural language processing tasks, including:
- Text generation and completion
- Summarization
- Question answering
- General conversational AI applications