automerger/PasticheInex12-7B
automerger/PasticheInex12-7B is a 7 billion parameter language model created by automerger, an automated merge of CorticalStack/pastiche-crown-clown-7b-dare and MSL7/INEX12-7b. This model leverages a slerp merge method across 32 layers of its constituent models, with specific parameter weighting for self-attention and MLP blocks. It is designed for general text generation tasks, combining characteristics from its merged components.
Loading preview...
PasticheInex12-7B: An Automated Merge Model
PasticheInex12-7B is a 7 billion parameter language model resulting from an automated merge process. Developed by automerger, this model combines the strengths of two base models: CorticalStack/pastiche-crown-clown-7b-dare and MSL7/INEX12-7b.
Key Characteristics
- Merge Method: Utilizes a slerp (spherical linear interpolation) merge method, which is effective for combining model weights.
- Layer-wise Merging: The merge was applied across all 32 layers of both base models, ensuring a comprehensive integration of their architectures.
- Configurable Parameters: Specific weighting parameters (
tvalues) were applied to different components, with distinct values for self-attention and MLP blocks, indicating a fine-tuned approach to combining their functionalities. - Base Model: The merge used
CorticalStack/pastiche-crown-clown-7b-dareas its base, suggesting its architectural foundation.
Usage and Application
This model is suitable for various text generation tasks, benefiting from the combined capabilities of its merged predecessors. Developers can easily integrate it into their Python projects using the transformers library, as demonstrated in the provided usage example. Its 7B parameter size makes it a versatile option for applications requiring a balance between performance and computational resources.