flemmingmiguel/MarcMistral-7B
MarcMistral-7B is a 7 billion parameter language model developed by flemmingmiguel, created by merging nfaheem/Marcoroni-7b-DPO-Merge and EmbeddedLLM/Mistral-7B-Merge-14-v0.5. This experimental merge aims to combine models with high MMLU and ARC benchmark performance. It is designed to explore optimal base merges for further fine-tuning, focusing on retaining or improving specific qualities from its component models.
Loading preview...
MarcMistral-7B Overview
MarcMistral-7B is an experimental 7 billion parameter language model developed by flemmingmiguel. It is constructed as a merge of two distinct models: nfaheem/Marcoroni-7b-DPO-Merge and EmbeddedLLM/Mistral-7B-Merge-14-v0.5, utilizing the LazyMergekit framework. This merge strategy is an ongoing experiment to identify the most effective base model combination for subsequent fine-tuning efforts.
Key Characteristics
- Experimental Merge: Designed to test the synergy between models excelling in different benchmarks, specifically combining a model with high MMLU (Massive Multitask Language Understanding) performance with one strong in ARC (AI2 Reasoning Challenge).
- Component Models: Built upon
Marcoroni-7b-DPO-MergeandMistral-7B-Merge-14-v0.5, aiming to leverage their respective strengths. - Configuration: The merge uses a
slerpmethod, with specific parameter weighting for self-attention and MLP layers, and a fallback value for other tensors.
Intended Use
This model is primarily intended for researchers and developers looking to:
- Explore Model Merging: Investigate the effects of combining different pre-trained models to achieve specific performance profiles.
- Base for Fine-tuning: Serve as a foundational model for further domain-specific or task-specific fine-tuning, with the goal of identifying a "clear winner" in benchmarks among various experimental merges.