Moza-7B-v1.0: A Merged Language Model
Moza-7B-v1.0 is a 7 billion parameter language model developed by kidyu, built upon the mistralai/Mistral-7B-v0.1 base. It was created using the DARE TIES merge method, combining nine different pre-trained models. The merge process assigned higher weights to models such as NeuralHermes, OpenOrca, and neural-chat based on a "vibes" selection.
Key Characteristics
- Architecture: Based on Mistral-7B-v0.1, leveraging a DARE TIES merge.
- Merged Models: Incorporates diverse models including
cognitivecomputations/dolphin-2.2.1-mistral-7b, Open-Orca/Mistral-7B-OpenOrca, openchat/openchat-3.5-0106, mlabonne/NeuralHermes-2.5-Mistral-7B, GreenNode/GreenNode-mini-7B-multilingual-v1olet, berkeley-nest/Starling-LM-7B-alpha, viethq188/LeoScorpius-7B-Chat-DPO, meta-math/MetaMath-Mistral-7B, and Intel/neural-chat-7b-v3-3. - Prompt Format: Utilizes the
Alpaca instruction format for inference. - Quantized Version: A GGUF quantized version is available.
Performance Highlights
Evaluated on the Open LLM Leaderboard, Moza-7B-v1.0 achieved an average score of 69.66. Notable scores include:
- AI2 Reasoning Challenge (25-Shot): 66.55
- HellaSwag (10-Shot): 83.45
- MMLU (5-Shot): 62.77
- TruthfulQA (0-shot): 65.16
- GSM8k (5-shot): 62.55
This model is suitable for users looking for a versatile 7B model derived from a diverse set of fine-tuned bases, offering a balance across various reasoning and language understanding tasks.