Overview

CultriX/SeQwence-14Bv1 is a 14.8 billion parameter language model created by CultriX using the DARE TIES merge method. It is based on the Qwen/Qwen2.5-14B architecture and integrates capabilities from five distinct pre-trained models to achieve a balanced and robust performance profile. The merge process involved specific weighting and density parameters for each contributing model, optimizing for a broad range of tasks while maintaining memory and computational efficiency through int8 masking and bfloat16 dtype.

Key Capabilities

• Enhanced Reasoning: Incorporates strengths from models excelling in GPQA, MUSR, and MMLU-PRO benchmarks.
• Advanced Mathematical Skills: Benefits from models with exceptional IFEval and MATH Level 5 capabilities.
• Creative and Narrative Generation: Includes contributions designed to enhance creative and narrative task performance.
• Generalization and Data Diversity: Leverages diverse data and generalization from its constituent models.

Good For

• Applications requiring strong performance across a variety of benchmarks, including reasoning and mathematical problem-solving.
• Use cases demanding a balance of factual accuracy, logical inference, and creative text generation.
• Developers looking for a versatile 14B parameter model with a 32768-token context length, optimized for efficiency.