NeoCortex-7B-slerp Overview

NeoCortex-7B-slerp is a 7 billion parameter language model developed by Kukedlc. It is a product of merging two distinct models: Kukedlc/Neural4gsm8k and macadeliccc/WestLake-7B-v2-laser-truthy-dpo. This merge was performed using the slerp (spherical linear interpolation) method, a technique often employed to combine the weights of different models to achieve a blend of their respective strengths.

Key Characteristics

• Architecture: A merged model combining two 7B parameter base models.
• Merge Method: Utilizes slerp for weight interpolation, specifically configured with varying t values for self-attention and MLP layers.
• Base Models: Integrates Kukedlc/Neural4gsm8k and macadeliccc/WestLake-7B-v2-laser-truthy-dpo to inherit their capabilities.
• Context Length: Supports a context window of 4096 tokens.

Usage

This model can be easily integrated into Python applications using the transformers library. It supports standard text generation tasks, with example code provided for setting up a pipeline and generating responses from user prompts. The configuration details highlight the specific layer ranges and parameter values used during the slerp merge process, indicating a tailored approach to combining the source models.