liminerity/Mini-blurstral
TEXT GENERATIONPricing:Input $0.4 / Cached $0.08 / Output $0.8Concurrent Unit Cost:1Model Size:7BQuant:FP8Context Size:4kTool Calling:SupportedPublished:Jan 17, 2024License:apache-2.0Architecture:Transformer Open Weights Featherless Exclusive Cold
Mini-blurstral by liminerity is a 7 billion parameter language model, merged from Mistral-7B-v0.1 and Blur-7b-slerp-v0.1 using LazyMergekit. This model leverages a slerp merge method with specific parameter weighting for self_attn and mlp layers, aiming to combine the strengths of its base models. With a 4096-token context length, it is suitable for general text generation tasks where a balance of performance and efficiency is desired.
Loading preview...
Mini-blurstral Overview
Mini-blurstral is a 7 billion parameter language model developed by liminerity. It is a product of merging two distinct models: mistralai/Mistral-7B-v0.1 and liminerity/Blur-7b-slerp-v0.1. This merge was performed using LazyMergekit, a tool designed for combining different language models.
Key Characteristics
- Architecture: Based on the Mistral architecture, inheriting its foundational capabilities.
- Merge Method: Utilizes a
slerp(spherical linear interpolation) merge method, which is applied to specific layers and components of the base models. - Parameter Weighting: The merge configuration includes distinct weighting for
self_attnandmlplayers, suggesting an intentional blend to optimize certain aspects of the model's behavior. - Base Model: The merging process was anchored on
OpenPipe/mistral-ft-optimized-1218as the base model. - Precision: Configured to use
bfloat16data type for efficient computation.
Good For
- General Text Generation: Suitable for a wide range of text generation tasks, leveraging the combined strengths of its constituent models.
- Exploration of Merged Models: Provides a practical example of how model merging can be used to create new models with potentially enhanced or specialized characteristics from existing ones.