allknowingroger/LimmyAutomerge-7B-slerp
LimmyAutomerge-7B-slerp is a 7 billion parameter language model created by allknowingroger, formed by a slerp merge of automerger/MeliodasNeuralsirkrishna-7B and liminerity/M7-7b. This model leverages a specific layer-wise merging strategy to combine the strengths of its constituent models. It is designed for general text generation tasks, offering a balanced performance derived from its merged architecture.
Loading preview...
Model Overview
LimmyAutomerge-7B-slerp is a 7 billion parameter language model developed by allknowingroger. It is constructed using a slerp (spherical linear interpolation) merge method via LazyMergekit, combining two distinct base models: automerger/MeliodasNeuralsirkrishna-7B and liminerity/M7-7b.
Merge Configuration
The merge process specifically targets all 32 layers of both source models. A key aspect of this merge is the differentiated t parameter application:
- Self-attention layers: The merge ratio
tvaries across layers, with values like[0, 0.5, 0.3, 0.7, 1]applied. - MLP layers: The merge ratio
talso varies, using values such as[1, 0.5, 0.7, 0.3, 0]. - Other parameters: A default
tvalue of0.5is applied.
This fine-grained control over the merge ratios for different layer types aims to optimize the combined model's performance characteristics. The model uses bfloat16 for its data type.
Usage
Developers can integrate LimmyAutomerge-7B-slerp using the Hugging Face transformers library. The provided Python example demonstrates how to load the model and tokenizer, apply a chat template, and generate text with specified parameters like max_new_tokens, temperature, top_k, and top_p.
Top 3 parameter combinations used by Featherless users for this model. Click a tab to see each config.