ajtaltarabukin2022/merged_beat_champ_3model_ties
The ajtaltarabukin2022/merged_beat_champ_3model_ties is a 32 billion parameter language model created by ajtaltarabukin2022 using the TIES merge method. It combines three pre-trained affine models, leveraging specific weighting for each component. This model is designed for general language tasks, inheriting capabilities from its merged constituents.
Loading preview...
Model Overview
ajtaltarabukin2022/merged_beat_champ_3model_ties is a 32 billion parameter language model developed by ajtaltarabukin2022. This model was constructed using the TIES (Trimming and Expanding) merge method, a technique designed to combine multiple pre-trained language models into a single, more capable model. The merging process utilized mergekit and was based on dura-lori/affine-5DoKPQhZmKnFk4mNEmH4UorbqHDe3PFAPvEfJyDwNkimoAMe as the base model.
Merge Details
This model integrates three distinct pre-trained affine models, with specific weights assigned to each during the merge:
- Base Model:
dura-lori/affine-5DoKPQhZmKnFk4mNEmH4UorbqHDe3PFAPvEfJyDwNkimoAMe(weighted at 0.45) - Component 1:
RLStepone/Affine-h29-5Coip2NhkPhFCMLQ7LYs3zLVz9RSEZP7HJrakDeqM5RVdPs4(weighted at 0.3) - Component 2:
fakemoonlo/Affine-5FnfLT3ntQXDsAnVC5H5WNQYVTY7SSCbxU3kxqhNybtJeNGb(weighted at 0.25)
All models were merged using bfloat16 data type, and the tokenizer from the base model was retained. This approach aims to consolidate the strengths of the individual models into a unified architecture.