ikimyaii/llama-7b-wanda-70pct
The ikimyaii/llama-7b-wanda-70pct model is a 7 billion parameter LLaMA-based language model, specifically an OpenLLaMA-7B variant, that has been pruned to 70% weight sparsity using the Wanda method. This semi-structured pruning removes the lowest-scoring weights in each output row based on magnitude and input-channel activation RMS, without weight correction or retraining. It serves as a research checkpoint for studying one-shot pruning techniques and is a baseline in the CWS comparison, demonstrating the impact of high sparsity on model performance with a WikiText-2 PPL of 82.10 at 70% sparsity.
Loading preview...
Overview
This model, ikimyaii/llama-7b-wanda-70pct, is a 7 billion parameter LLaMA-based language model derived from openlm-research/open_llama_7b. It has undergone one-shot pruning to 70% weight sparsity using the Wanda method, a technique that scores weights by their magnitude and input-channel activation RMS, removing the lowest-scoring ones in each output row. This process is applied to every nn.Linear layer in the 32 decoder layers and the lm_head, totaling 225 pruned layers, while embeddings and norms remain dense.
Key Capabilities
- Research Checkpoint: Primarily designed for studying the effects of high weight sparsity (70%) on large language models without retraining.
- Wanda Pruning Method: Demonstrates the application and performance characteristics of the Wanda pruning technique (Sun et al., ICLR 2024, arXiv:2306.11695).
- Baseline Comparison: Serves as a comparative baseline within the CWS project for evaluating different pruning strategies.
Good for
- Sparsity Research: Ideal for researchers investigating one-shot pruning, weight sparsity, and their impact on LLM performance.
- Methodology Study: Understanding the Wanda pruning method and its effectiveness at high sparsity levels.
- Comparative Analysis: Using as a benchmark against other pruning techniques like CWS, SparseGPT, RIA, and AWP, particularly for WikiText-2 PPL evaluation.
Limitations
It's important to note that these are research checkpoints and are not instruction-tuned or safety-tuned. At 70-80% sparsity, the model does not produce usable text and zero-shot accuracy is near chance. Weights are stored densely with zeros, meaning sparsity alone does not provide speedup without specialized sparse kernels or hardware.