appvoid/cloud-10
appvoid/cloud-10 is a 0.35 billion parameter language model created by appvoid using the DARE TIES merge method. It combines appvoid/palmer-005-core with three other 350M parameter models, resulting in a compact model with a 32768 token context length. This model is designed for general language tasks, leveraging the strengths of its merged components for efficient performance.
Loading preview...
Overview
appvoid/cloud-10 is a compact 0.35 billion parameter language model developed by appvoid. It was constructed using the DARE TIES merge method via mergekit, building upon appvoid/palmer-005-core as its base.
Merge Details
This model integrates capabilities from three distinct 350M parameter models:
MihaiPopa-1/LFM2.5-350M-hereticmkurman/LiquidAI-LFM2.5-350M-SYNTHsqu11z1/claude-oss-350m
The merge configuration utilized specific weighting and density parameters for each component model, with squ11z1/claude-oss-350m having the highest weight. The process involved bfloat16 data type and included int8_mask for parameter handling.
Key Characteristics
- Parameter Count: 0.35 billion parameters, making it suitable for resource-constrained environments.
- Context Length: Supports a substantial context window of 32768 tokens.
- Merge Method: Leverages the DARE TIES method for combining pre-trained models, aiming to consolidate their strengths.
Use Cases
This model is well-suited for applications requiring a smaller footprint while benefiting from the combined knowledge of its constituent models. Its design suggests applicability in general language understanding and generation tasks where efficiency and a reasonable context window are important.