ajtaltarabukin2022/merged_champion_v5_m4
The ajtaltarabukin2022/merged_champion_v5_m4 is a 32 billion parameter language model created by ajtaltarabukin2022, formed by merging multiple pre-trained models using the Linear merge method via mergekit. This model integrates contributions from dura-lori/affine-5ED5dwT4fztHjgjyR6vXpbGfnooeuWfr3VueaZrrfWJSou7y, voidai001/affine-rl0-5HeJuQB4ZcVaU8yfgwYCm3AvdiA7dPA34nvB5HwSubVoFREm, chouchouM/Affine-5DhGPvYiBChDerVjSgyt1vuuwQyZWJJgsEdQHAkXRuSYji4d, and dura-lori/affine-5CtqFaxMkR1rZfP3cWiW6ywTszxd6dKqFoPtKdLQzMkT1kCf, with a context length of 32768 tokens. Its primary differentiation lies in its construction as a composite model, leveraging the strengths of its constituent parts for general language understanding and generation tasks.
Loading preview...
Model Overview
This model, merged_champion_v5_m4, is a 32 billion parameter language model developed by ajtaltarabukin2022. It was created using the mergekit tool, specifically employing the Linear merge method.
Merge Details
The model's architecture is a composite, built upon a base model and integrating contributions from several other pre-trained models. The base model for this merge was dura-lori/affine-5ED5dwT4fztHjgjyR6vXpbGfnooeuWfr3VueaZrrfWJSou7y.
Constituent Models
The merge incorporated the following models, each contributing with specific weights across their layers:
- voidai001/affine-rl0-5HeJuQB4ZcVaU8yfgwYCm3AvdiA7dPA34nvB5HwSubVoFREm
- chouchouM/Affine-5DhGPvYiBChDerVjSgyt1vuuwQyZWJJgsEdQHAkXRuSYji4d
- dura-lori/affine-5CtqFaxMkR1rZfP3cWiW6ywTszxd6dKqFoPtKdLQzMkT1kCf
Configuration
The merge process utilized a bfloat16 data type and applied specific weighting parameters to each contributing model's layers (0 to 64) during the linear merge. This approach aims to combine the learned representations and capabilities of the individual models into a single, more robust model with a 32768 token context length.