CollectiveCognition-v1.1-Mistral-7B-dare-0.85 Overview

This model is an experimental 7 billion parameter Mistral-based language model developed by uukuguy, focusing on the DARE (Drop and REscale) technique. DARE investigates the possibility of setting a significant portion of delta parameters to zero without compromising the model's capabilities, particularly in fine-tuned language models and larger architectures.

Key Experimental Parameters

The model was trained with specific DARE configurations:

• Weight Mask Rate: 0.85 (meaning 85% of delta parameters are set to zero)
• Weight Rescaling: Enabled (use_weight_rescale: True)
• Masking Strategy: Random (mask_stratery: random)
• Scaling Coefficient: 1.0

Performance Insights

While the primary focus is on the DARE methodology, the model's performance is benchmarked against several other 7B models across various tasks, including ARC, HellaSwag, MMLU, TruthfulQA, Winogrande, GSM8K, and DROP. The provided benchmark table allows for a comparative analysis of its average performance relative to other established models in its class, indicating its general language understanding and reasoning capabilities within the experimental DARE framework.