PasticheAlloyingotneoy-7B: An Automated Merge Model
PasticheAlloyingotneoy-7B is a 7 billion parameter language model developed through an automated merging process by automerger, specifically configured by Maxime Labonne. This model is a strategic combination of two distinct base models: CorticalStack/pastiche-crown-clown-7b-dare-dpo and nlpguy/AlloyIngotNeoY.
Key Characteristics
- Automated Merging: The model is a product of an automated merge using a
slerp method, which intelligently combines the weights of its constituent models. - Layer-wise Integration: The merge process specifically targets layers 0 through 32 of both base models, allowing for fine-grained integration of their respective capabilities.
- Configurable Parameters: The merging process includes specific
t parameters for self_attn and mlp filters, indicating a tailored approach to weight distribution across different architectural components. - Bfloat16 Precision: The model is configured to use
bfloat16 data type, optimizing for both performance and memory efficiency.
Use Cases
This model is suitable for general text generation tasks, benefiting from the combined strengths of its merged predecessors. Its 7B parameter count and 4096-token context length make it a versatile option for various natural language processing applications.