automerger/PasticheAlloyingotneoy-7B

TEXT GENERATIONConcurrency Cost:1Model Size:7BQuant:FP8Ctx Length:4kPublished:Mar 11, 2024License:apache-2.0Architecture:Transformer Open Weights Cold

PasticheAlloyingotneoy-7B is a 7 billion parameter language model created by automerger, an automated merge of CorticalStack/pastiche-crown-clown-7b-dare-dpo and nlpguy/AlloyIngotNeoY. This model leverages a slerp merge method across its 32 layers, combining the strengths of its base models. It is designed for general text generation tasks, offering a 4096-token context length.

Loading preview...

PasticheAlloyingotneoy-7B: An Automated Merge Model

PasticheAlloyingotneoy-7B is a 7 billion parameter language model developed through an automated merging process by automerger, specifically configured by Maxime Labonne. This model is a strategic combination of two distinct base models: CorticalStack/pastiche-crown-clown-7b-dare-dpo and nlpguy/AlloyIngotNeoY.

Key Characteristics

  • Automated Merging: The model is a product of an automated merge using a slerp method, which intelligently combines the weights of its constituent models.
  • Layer-wise Integration: The merge process specifically targets layers 0 through 32 of both base models, allowing for fine-grained integration of their respective capabilities.
  • Configurable Parameters: The merging process includes specific t parameters for self_attn and mlp filters, indicating a tailored approach to weight distribution across different architectural components.
  • Bfloat16 Precision: The model is configured to use bfloat16 data type, optimizing for both performance and memory efficiency.

Use Cases

This model is suitable for general text generation tasks, benefiting from the combined strengths of its merged predecessors. Its 7B parameter count and 4096-token context length make it a versatile option for various natural language processing applications.

Finetunes
2 models