allknowingroger/TripleMerge2-7B-Ties
TripleMerge2-7B-Ties is a 7 billion parameter language model created by allknowingroger, formed by merging LimyQstar-7B-slerp, JaskierMistral-7B-slerp, and LimmyAutomerge-7B-slerp using the TIES merging method. This model leverages a density and weight gradient configuration across its merged components. It is designed for general text generation tasks, combining the strengths of its constituent models.
Loading preview...
Model Overview
TripleMerge2-7B-Ties is a 7 billion parameter language model developed by allknowingroger. It is constructed through a merge of three distinct models: LimyQstar-7B-slerp, JaskierMistral-7B-slerp, and LimmyAutomerge-7B-slerp. The merging process utilizes the TIES (Trimming, Iterative, and Selective) method, implemented via LazyMergekit.
Merging Configuration
The model's unique characteristics stem from its specific merging configuration, which applies density and weight gradients to its constituent models:
- LimyQstar-7B-slerp: Integrated with a density gradient from 1 to 0.1 and a weight of 1.0.
- JaskierMistral-7B-slerp: Incorporated with a fixed density of 0.5 and a weight gradient from 0 to 1.
- LimmyAutomerge-7B-slerp: Included with a density of 0.33, applying a conditional weight of 0.5 for MLP layers and 0 otherwise.
This intricate merging strategy aims to combine the strengths of the base models, with normalization and INT8 masking applied during the merge. The base model for the merge was LimyQstar-7B-slerp.
Usage
Developers can interact with TripleMerge2-7B-Ties using the Hugging Face transformers library, leveraging AutoTokenizer and pipeline for text generation tasks. The model supports standard chat template application for user messages.
Top 3 parameter combinations used by Featherless users for this model. Click a tab to see each config.