ikimyaii/llama-7b-wanda-30pct
The ikimyaii/llama-7b-wanda-30pct model is a 7 billion parameter LLaMA-based language model, specifically a pruned version of OpenLLaMA-7B. It achieves 30% weight sparsity using the Wanda one-shot pruning method, where weights are removed based on magnitude and input-channel activation RMS. This model serves as a research checkpoint for studying the effects of pruning on large language models, demonstrating competitive perplexity on WikiText-2 compared to other sparsity methods.
Loading preview...
Overview
The ikimyaii/llama-7b-wanda-30pct model is a 7 billion parameter variant of the OpenLLaMA-7B base model, distinguished by its significant weight sparsity. It has been pruned one-shot to approximately 30% weight sparsity using the Wanda method, which scores weights based on their magnitude and input-channel activation RMS. This process removes the lowest-scoring weights from each output row, specifically targeting nn.Linear layers within the decoder.
Key Characteristics
- Base Model: Derived from
openlm-research/open_llama_7b. - Pruning Method: Employs the Wanda method (Sun et al., ICLR 2024), which is a one-shot pruning technique without retraining or weight correction.
- Sparsity Level: Achieves 29.98% sparsity, with zeros at the pruned positions.
- Research Focus: Primarily serves as a research checkpoint for investigating pruning techniques and their impact on LLMs, particularly within the context of the CWS comparison framework.
Performance Insights
Evaluation on WikiText-2 shows a perplexity (PPL) of 9.04 for this Wanda-pruned model at 30% sparsity, compared to the dense model's 8.64 PPL. This places its performance competitively among other pruning methods like CWS (9.12 PPL), SparseGPT (9.18 PPL), RIA (8.96 PPL), and AWP (8.92 PPL) under the same evaluation protocol.
Limitations
It is important to note that this model is a research checkpoint and is not instruction-tuned or safety-tuned. It inherits the biases of its base model. Furthermore, the weights are stored densely with explicit zeros, meaning sparsity alone does not provide a speedup without specialized sparse kernels or event-driven hardware.