mergekit-community/Deutscher-Pantheon-12B
Deutscher-Pantheon-12B is a 12 billion parameter language model created by mergekit-community, built upon the Mistral-Nemo-Base-2407-chatml architecture. This model is a DARE TIES merge of Gryphe/Pantheon-RP-1.6-12b-Nemo and VAGOsolutions/SauerkrautLM-Nemo-12b-Instruct, combining their respective strengths. With a context length of 32768 tokens, it is designed to leverage the capabilities of its merged components for diverse language generation tasks.
Loading preview...
Deutscher-Pantheon-12B: A Merged Language Model
Deutscher-Pantheon-12B is a 12 billion parameter language model developed by mergekit-community. It was constructed using the DARE TIES merge method, a technique designed to combine the strengths of multiple pre-trained models efficiently. The base model for this merge is IntervitensInc/Mistral-Nemo-Base-2407-chatml, providing a robust foundation.
Key Components and Merge Strategy
This model integrates two distinct language models:
- Gryphe/Pantheon-RP-1.6-12b-Nemo: Contributes with a weight of 0.4 and a density of 0.5.
- VAGOsolutions/SauerkrautLM-Nemo-12b-Instruct: Contributes with a higher weight of 0.6 and a density of 0.5.
The DARE TIES method, as detailed in its respective research papers, allows for a nuanced combination of model parameters, aiming to preserve and enhance the capabilities of the constituent models. The merge was performed with bfloat16 data type precision.
Potential Use Cases
Given its merged nature, Deutscher-Pantheon-12B is likely suitable for applications that benefit from the combined strengths of its source models. Users looking for a model that integrates the characteristics of both Pantheon-RP and SauerkrautLM, built on a Mistral-Nemo base, may find this model particularly useful for general language generation and instruction-following tasks.