ikimyaii/llama-7b-cws-60pct
The ikimyaii/llama-7b-cws-60pct model is a 7 billion parameter LLaMA-based language model, specifically OpenLLaMA-7B, that has been pruned to 60% weight sparsity using the Correlation-aware Weight Sparsity (CWS) method. This semi-structured pruning technique, which extends SparseGPT's Optimal Brain Surgeon with a full-Hessian selection criterion, aims to maintain performance while significantly reducing model size. It is primarily designed for research into efficient LLM inference and studying the effects of post-training one-shot pruning.
Loading preview...
Overview
ikimyaii/llama-7b-cws-60pct is a 7 billion parameter language model based on openlm-research/open_llama_7b. This model has undergone one-shot pruning to 60% weight sparsity using the Correlation-aware Weight Sparsity (CWS) method. CWS is a post-training pruning technique that extends SparseGPT's Optimal Brain Surgeon (OBS) by incorporating a full-Hessian selection criterion, allowing it to account for cancellation groups and cross-weight interactions that simpler methods miss. The pruning is applied to every nn.Linear layer in the 32 decoder layers and lm_head, resulting in 225 pruned layers, while embeddings and norms remain dense.
Key Capabilities
- High Sparsity: Achieves 60% weight sparsity, significantly reducing the number of active parameters.
- Advanced Pruning Method: Utilizes CWS, which considers correlation between weights for more effective pruning compared to methods relying solely on individual weight saliency scores.
- Semi-Structured Pruning: Removes the same fraction of weights from every output row, making it suitable for hardware acceleration that supports semi-structured sparsity.
- Research Checkpoint: Serves as a valuable resource for studying the impact of advanced pruning techniques on large language models.
Good For
- Research on LLM Pruning: Ideal for researchers investigating post-training quantization and sparsity methods for large language models.
- Exploring CWS: Provides a practical example of the CWS pruning method in action on a LLaMA-based architecture.
- Benchmarking Sparse Models: Can be used to compare the performance and efficiency of CWS against other pruning techniques like SparseGPT, Wanda, RIA, and AWP.
Limitations
It's important to note that these are research checkpoints. The model is not instruction-tuned or safety-tuned and inherits the biases of its base model. Additionally, weights are stored densely with zeros at pruned positions, meaning sparsity alone does not provide speedup without specialized sparse kernels or event-driven hardware.