ikimyaii/llama-7b-cws-80pct
The ikimyaii/llama-7b-cws-80pct model is a 7 billion parameter LLaMA-based language model, specifically OpenLLaMA-7B, that has been pruned to 80% weight sparsity using the Correlation-aware Weight Sparsity (CWS) method. This model is a research checkpoint designed for studying post-training one-shot pruning techniques, focusing on the CWS method which extends SparseGPT's Optimal Brain Surgeon with a full-Hessian selection criterion. Its primary use case is for research into efficient LLM inference and understanding the effects of high sparsity on model performance, rather than for generating usable text.
Loading preview...
Overview
ikimyaii/llama-7b-cws-80pct is a 7 billion parameter model derived from openlm-research/open_llama_7b. It has been pruned to approximately 80% weight sparsity using the Correlation-aware Weight Sparsity (CWS) method. CWS is a post-training, one-shot pruning technique that enhances SparseGPT's Optimal Brain Surgeon (OBS) by incorporating a full-Hessian selection criterion. This allows it to account for cancellation groups and cross-weight interactions that simpler saliency scores miss, aiming for more effective pruning.
Key Capabilities
- High Sparsity Research: Demonstrates the application of CWS to achieve 80% weight sparsity in a 7B LLaMA model.
- Advanced Pruning Method: Utilizes CWS, which employs a greedy selection process within blocks (Block CWS) and applies exact OBS corrections to surviving weights.
- Detailed Pruning Scope: Pruning is applied to
nn.Linearlayers within the 32 decoder layers (attention q/k/v/o and MLP gate/up/down) andlm_head, totaling 225 layers, while embeddings and norms remain dense.
Good For
- Sparsity Research: Ideal for researchers investigating advanced post-training pruning methods like CWS and their impact on large language models.
- Methodology Comparison: Useful for comparing CWS against other pruning techniques such as SparseGPT, Wanda, RIA, and AWP, as evaluation metrics are provided.
- Understanding Pruning Effects: Provides a practical example of a highly sparse model for analyzing the trade-offs between sparsity and performance, particularly regarding perplexity and zero-shot accuracy on benchmarks like WikiText-2, ARC, HellaSwag, PIQA, and WinoGrande.
Note: This model is a research checkpoint and is not instruction-tuned or safety-tuned. At 70-80% sparsity, it does not produce usable text, and its zero-shot accuracy is near chance, indicating its primary purpose is for studying pruning techniques rather than practical application.