brucethemoose/Capybara-Tess-Yi-34B-200K-DARE-Ties
brucethemoose/Capybara-Tess-Yi-34B-200K-DARE-Ties is a 34 billion parameter language model based on the Yi architecture, featuring a 32K context length. This model is an experimental merge of Nous-Capybara-34B, Tess-M-v1.2, and Tess-M-v1.3 using the DARE Ties method, designed to absorb abilities from its constituent models. It aims to provide a more effective merge than traditional linear or slerp methods, leveraging different training datasets from the Tess models.
Loading preview...
Model Overview
Capybara-Tess-Yi-34B-200K-DARE-Ties is an experimental 34 billion parameter language model created by brucethemoose. It is a merge of three distinct models: NousResearch/Nous-Capybara-34B, migtissera/Tess-M-v1.2, and migtissera/Tess-M-v1.3. The merge was performed using a novel implementation of the "DARE Ties" method, as described in the paper "Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch" (https://github.com/yule-BUAA/MergeLM). This approach aims to yield a more effective combination of model strengths compared to standard merging techniques.
Key Characteristics
- DARE Ties Merge: Utilizes an experimental "dare_ties" merge method via
mergekit(https://github.com/cg123/mergekit/tree/dare-tokenizer). - Diverse Base Models: Incorporates models trained on different datasets, particularly Tess-M-v1.2 and Tess-M-v1.3, to potentially enhance overall capabilities.
- Yi Architecture: Built upon a Yi-34B-200K-Llamafied base model, supporting a 32K context length.
- Prompt Format: Designed to work with the Orca-Vicuna prompt template.
Usage Notes
- Users may need to disable the BOS token or use a lower temperature with MinP for optimal output, characteristic of Yi models.
- The model may sometimes spell out the stop token as
</s>, requiring</s>to be added as an additional stopping condition.