automerger/Inex12Yamshadow-7B
TEXT GENERATIONConcurrency Cost:1Model Size:7BQuant:FP8Ctx Length:4kPublished:Mar 15, 2024License:apache-2.0Architecture:Transformer Open Weights Cold
automerger/Inex12Yamshadow-7B is a 7 billion parameter language model created by Maxime Labonne through an automated merge process. This model combines MSL7/INEX12-7b and automerger/YamShadow-7B using a slerp merge method, specifically adjusting parameters for self_attn and mlp layers. It is designed for general language generation tasks, leveraging the combined strengths of its constituent models within a 4096-token context length.
Loading preview...
Overview
automerger/Inex12Yamshadow-7B is a 7 billion parameter language model developed by Maxime Labonne. It is an automated merge of two distinct models: MSL7/INEX12-7b and automerger/YamShadow-7B.
Key Capabilities
- Automated Merge Architecture: Utilizes a
slerp(spherical linear interpolation) merge method to combine the weights of its base models. - Parameter Specific Merging: The merge configuration applies varying
tvalues forself_attnandmlplayers, suggesting a fine-tuned approach to integrate the strengths of both source models. - General Language Generation: Suitable for a wide range of text generation tasks, leveraging its 7 billion parameters and 4096-token context window.
Good For
- Experimentation with Merged Models: Developers interested in exploring the performance characteristics of models created via automated merging techniques.
- General Purpose Text Generation: Can be used for various applications requiring text completion, question answering, or content creation based on its foundational models.
- Understanding Merge Strategies: Provides a practical example of a
slerpmerge with specific layer-wise parameter adjustments.