Kukedlc/NeoCortex-7B-slerp
Kukedlc/NeoCortex-7B-slerp is a 7 billion parameter language model created by Kukedlc, formed by merging Kukedlc/Neural4gsm8k and macadeliccc/WestLake-7B-v2-laser-truthy-dpo using a slerp method. This model leverages the strengths of its base components, with a context length of 4096 tokens. It is designed for general language generation tasks, benefiting from the combined capabilities of its merged predecessors.
Loading preview...
NeoCortex-7B-slerp Overview
NeoCortex-7B-slerp is a 7 billion parameter language model developed by Kukedlc. It is a product of merging two distinct models: Kukedlc/Neural4gsm8k and macadeliccc/WestLake-7B-v2-laser-truthy-dpo. This merge was performed using the slerp (spherical linear interpolation) method, a technique often employed to combine the weights of different models to achieve a blend of their respective strengths.
Key Characteristics
- Architecture: A merged model combining two 7B parameter base models.
- Merge Method: Utilizes
slerpfor weight interpolation, specifically configured with varyingtvalues for self-attention and MLP layers. - Base Models: Integrates
Kukedlc/Neural4gsm8kandmacadeliccc/WestLake-7B-v2-laser-truthy-dpoto inherit their capabilities. - Context Length: Supports a context window of 4096 tokens.
Usage
This model can be easily integrated into Python applications using the transformers library. It supports standard text generation tasks, with example code provided for setting up a pipeline and generating responses from user prompts. The configuration details highlight the specific layer ranges and parameter values used during the slerp merge process, indicating a tailored approach to combining the source models.