Model Overview

The speechless-mistral-six-in-one-7b-orth-1.0 is a 7 billion parameter language model built upon the Mistral-7B architecture. Developed by uukuguy, this model is a unique merge of six prominent Mistral-7B based models, including ehartford/dolphin-2.1-mistral-7b, Open-Orca/Mistral-7B-OpenOrca, and HuggingFaceH4/zephyr-7b-alpha, among others.

Key Differentiator

What sets this model apart is its "orthogonal modification" approach during fine-tuning. This method adjusts the base model weights by considering only those changes that are orthogonal to the original weight direction. The goal is to capture the essence of fine-tuning improvements while maintaining the foundational structure of the original model as much as possible.

Performance Highlights

While specific HumanEval scores for this exact model are not provided, the base Mistral-7B-v0.1 scores 30.488. The merged speechless-mistral-six-in-one-7b (the precursor to this orthogonal version) was rated highly by a community benchmark, achieving an average of 98/100 across intellect, creativity, adaptability, communication, and problem-solving when compared against LLaMa2 70B chat. It also shows competitive results on the LM-Evaluation-Harness, with an average score of 53.38, including 63.29 on MMLU and 84.6 on HellaSwag.

Use Cases

This model is suitable for a wide range of general-purpose language generation and understanding tasks, particularly where a balance of intellectual capability, creativity, and adaptability is desired within a 7B parameter constraint. Its unique merging and orthogonal fine-tuning strategy aim to provide a robust and versatile foundation for various applications.