ikimyaii/llama-7b-wanda-60pct
The ikimyaii/llama-7b-wanda-60pct model is a 7 billion parameter LLaMA-based language model, specifically OpenLLaMA-7B, that has been pruned to 60% weight sparsity using the Wanda method. This one-shot pruning technique removes the lowest-scoring weights in each output row based on magnitude and input-channel activation RMS, without retraining or weight correction. It serves as a research checkpoint for studying pruning techniques, demonstrating competitive perplexity on WikiText-2 compared to other sparsity methods, and is intended for research into efficient model architectures.
Loading preview...
Overview
ikimyaii/llama-7b-wanda-60pct is a 7 billion parameter language model derived from openlm-research/open_llama_7b. This model has undergone one-shot pruning to achieve 60% weight sparsity using the Wanda method. Wanda (Sun et al., ICLR 2024) scores weights by their magnitude multiplied by the input-channel activation RMS, removing the lowest-scoring weights in each output row without any weight correction or retraining. This makes it a valuable research checkpoint for exploring efficient model architectures and sparsity techniques.
Key Capabilities
- High Sparsity: Achieves approximately 60% weight sparsity, significantly reducing the number of active parameters.
- Efficient Pruning Method: Utilizes the Wanda method, which is a one-shot pruning technique requiring no retraining.
- Competitive Performance: Demonstrates a WikiText-2 perplexity (PPL) of 18.11 at 60% sparsity, outperforming SparseGPT and RIA at the same sparsity level.
- Research Baseline: Serves as a baseline model in the CWS comparison for evaluating various pruning strategies.
Good For
- Sparsity Research: Ideal for researchers studying weight pruning, model compression, and the effects of sparsity on LLM performance.
- Benchmarking: Useful for comparing different pruning methods and their impact on perplexity and model efficiency.
- Understanding Pruning Techniques: Provides a practical example of the Wanda pruning method applied to a LLaMA-based model.
Limitations
It is important to note that this model is a research checkpoint. It is not instruction-tuned or safety-tuned and inherits the biases of its base model. Additionally, the weights are stored densely with zeros, meaning sparsity alone does not provide speedup without specialized sparse kernels or hardware.