PasticheInex12-7B: An Automated Merge Model

PasticheInex12-7B is a 7 billion parameter language model resulting from an automated merge process. Developed by automerger, this model combines the strengths of two base models: CorticalStack/pastiche-crown-clown-7b-dare and MSL7/INEX12-7b.

Key Characteristics

• Merge Method: Utilizes a slerp (spherical linear interpolation) merge method, which is effective for combining model weights.
• Layer-wise Merging: The merge was applied across all 32 layers of both base models, ensuring a comprehensive integration of their architectures.
• Configurable Parameters: Specific weighting parameters (t values) were applied to different components, with distinct values for self-attention and MLP blocks, indicating a fine-tuned approach to combining their functionalities.
• Base Model: The merge used CorticalStack/pastiche-crown-clown-7b-dare as its base, suggesting its architectural foundation.

Usage and Application

This model is suitable for various text generation tasks, benefiting from the combined capabilities of its merged predecessors. Developers can easily integrate it into their Python projects using the transformers library, as demonstrated in the provided usage example. Its 7B parameter size makes it a versatile option for applications requiring a balance between performance and computational resources.