Model Overview

TripleMerge2-7B-Ties is a 7 billion parameter language model developed by allknowingroger. It is constructed through a merge of three distinct models: LimyQstar-7B-slerp, JaskierMistral-7B-slerp, and LimmyAutomerge-7B-slerp. The merging process utilizes the TIES (Trimming, Iterative, and Selective) method, implemented via LazyMergekit.

Merging Configuration

The model's unique characteristics stem from its specific merging configuration, which applies density and weight gradients to its constituent models:

• LimyQstar-7B-slerp: Integrated with a density gradient from 1 to 0.1 and a weight of 1.0.
• JaskierMistral-7B-slerp: Incorporated with a fixed density of 0.5 and a weight gradient from 0 to 1.
• LimmyAutomerge-7B-slerp: Included with a density of 0.33, applying a conditional weight of 0.5 for MLP layers and 0 otherwise.

This intricate merging strategy aims to combine the strengths of the base models, with normalization and INT8 masking applied during the merge. The base model for the merge was LimyQstar-7B-slerp.

Usage

Developers can interact with TripleMerge2-7B-Ties using the Hugging Face transformers library, leveraging AutoTokenizer and pipeline for text generation tasks. The model supports standard chat template application for user messages.