uukuguy/CollectiveCognition-v1.1-Mistral-7B-dare-0.85
uukuguy/CollectiveCognition-v1.1-Mistral-7B-dare-0.85 is a 7 billion parameter Mistral-based language model developed by uukuguy, utilizing the DARE (Drop and REscale) experimental technique. This model explores the impact of setting a high proportion of delta parameters to zero, specifically with a 0.85 weight mask rate. It aims to maintain capabilities while potentially reducing model complexity, offering a unique approach to model optimization.
Loading preview...
CollectiveCognition-v1.1-Mistral-7B-dare-0.85 Overview
This model is an experimental 7 billion parameter Mistral-based language model developed by uukuguy, focusing on the DARE (Drop and REscale) technique. DARE investigates the possibility of setting a significant portion of delta parameters to zero without compromising the model's capabilities, particularly in fine-tuned language models and larger architectures.
Key Experimental Parameters
The model was trained with specific DARE configurations:
- Weight Mask Rate: 0.85 (meaning 85% of delta parameters are set to zero)
- Weight Rescaling: Enabled (
use_weight_rescale: True) - Masking Strategy: Random (
mask_stratery: random) - Scaling Coefficient: 1.0
Performance Insights
While the primary focus is on the DARE methodology, the model's performance is benchmarked against several other 7B models across various tasks, including ARC, HellaSwag, MMLU, TruthfulQA, Winogrande, GSM8K, and DROP. The provided benchmark table allows for a comparative analysis of its average performance relative to other established models in its class, indicating its general language understanding and reasoning capabilities within the experimental DARE framework.