uukuguy/zephyr-7b-alpha-dare-0.85
uukuguy/zephyr-7b-alpha-dare-0.85 is a 7 billion parameter experimental language model based on the Zephyr-7B-alpha architecture, fine-tuned using the DARE (Drop and REscale) method. This model explores the concept that a high proportion (85%) of delta parameters can be set to zero without significantly impacting capabilities. It is designed to investigate parameter efficiency and model compression techniques, maintaining an 8192-token context length.
Loading preview...
uukuguy/zephyr-7b-alpha-dare-0.85: An Experiment in Parameter Efficiency
This model, developed by uukuguy, is a 7 billion parameter variant of the Zephyr-7B-alpha architecture, specifically fine-tuned using the DARE (Drop and REscale) experimental method. The core idea behind DARE is to demonstrate that a significant portion of delta parameters in fine-tuned language models can be discarded (set to zero) without compromising performance.
Key Characteristics
- Parameter Efficiency Experiment: Investigates the DARE method with a
weight_mask_rateof 0.85, meaning 85% of delta parameters are masked. - Rescaling: Utilizes
use_weight_rescale: Trueand ascaling_coefficient: 1.0to compensate for the dropped parameters. - Base Model: Built upon the HuggingFaceH4/zephyr-7b-alpha foundation.
- Context Length: Supports an 8192-token context window.
Performance Insights
The model's performance is evaluated across various benchmarks, including ARC, HellaSwag, MMLU, TruthfulQA, Winogrande, GSM8K, and DROP. While not leading in all categories, its average score of 52.4 on the provided benchmark table is comparable to the base Zephyr-7B-alpha model (52.4 average), suggesting that the DARE method at an 85% mask rate can maintain performance levels. This makes it a valuable resource for research into model compression and efficient fine-tuning techniques.