Model Overview
Collective Cognition v1.1-Mistral-7B is a 7 billion parameter model developed by teknium, fine-tuned using the Mistral architecture. A key differentiator for this model is its remarkable performance on the TruthfulQA benchmark, where it competes strongly with and often outperforms larger 70B models. This achievement is particularly notable given its rapid training time (3 minutes on a single 4090 GPU using QLoRA) and its training on a very small dataset of only 100 data points, sourced from a platform similar to ShareGPT.
Key Capabilities & Features
- Exceptional Truthfulness: Demonstrates high accuracy on the TruthfulQA benchmark, indicating a strong ability to avoid common misconceptions and reduce hallucination.
- Efficient Training: Achieves competitive performance with minimal computational resources and a highly limited dataset.
- Mistral-based Fine-tuning: Leverages the Mistral approach for its underlying architecture.
Performance Highlights
- TruthfulQA: Outperforms various 70B models, highlighting its strength in factual accuracy and misconception rectification.
- GPT4All Benchmarks: Achieves an average score of 71.13 across tasks like arc_challenge, arc_easy, boolq, hellaswag, openbookqa, piqa, and winogrande.
- AGIEval Benchmarks: Achieves an average score of 33.57 across various reasoning and comprehension tasks.
Ideal Use Cases
This model is particularly well-suited for applications where factual accuracy and the reduction of hallucinations are critical, especially in scenarios where computational resources or extensive training data are limited. Its strong performance on TruthfulQA suggests its utility in question-answering systems, content generation requiring high veracity, and educational tools.