Sorihon/Mending-Hearts-12B

Mending-Hearts-12B Overview

Mending-Hearts-12B is a 12 billion parameter language model developed by Sorihon, created through a sophisticated merging process. This model utilizes the DARE TIES merge method, a technique designed to combine the strengths of multiple pre-trained language models into a single, more capable entity.

Merge Details

The foundational model for this merge is djuna/MN-Chinofun-12B-4. It was combined with two additional models:

• Vortex5/Nether-Moon-12B
• MarinaraSpaghetti/NemoMix-Unleashed-12B

The DARE TIES method, as described in the original paper, allows for selective integration of parameters, optimizing for a balanced performance profile. The specific configuration involved varying density and weight parameters for each contributing model, with the entire process conducted in bfloat16 precision.

Key Characteristics

• Merged Architecture: Benefits from the combined knowledge and capabilities of three distinct 12B models.
• DARE TIES Method: Employs an advanced merging technique for potentially superior performance compared to simpler merges.
• Parameter Efficiency: A 12B parameter model offering a balance between performance and computational requirements.

Potential Use Cases

This model is suitable for developers looking for a robust 12B model that integrates diverse linguistic patterns and knowledge bases. Its merged nature suggests potential for broad applicability in general-purpose text generation, understanding, and conversational AI tasks where the strengths of its constituent models are beneficial.