ikimyaii/llama-7b-cws-30pct
The ikimyaii/llama-7b-cws-30pct model is a 7 billion parameter language model based on OpenLLaMA-7B, developed by ikimyaii. It has been pruned to 30% weight sparsity using the Correlation-aware Weight Sparsity (CWS) method, which extends SparseGPT with a full-Hessian selection criterion for more effective pruning. This model is a research checkpoint designed for studying post-training, one-shot pruning techniques, offering insights into efficient LLM inference by reducing model size while maintaining performance.
Loading preview...
ikimyaii/llama-7b-cws-30pct: A Research Model for Efficient LLM Pruning
This model is a 7 billion parameter variant of OpenLLaMA-7B, developed by ikimyaii, that has been pruned to approximately 30% weight sparsity. It utilizes the Correlation-aware Weight Sparsity (CWS) method, a novel post-training, one-shot pruning technique. CWS enhances SparseGPT's Optimal Brain Surgeon (OBS) by incorporating a full-Hessian selection criterion, which accounts for cross-weight interactions and cancellation groups that simpler saliency scores miss.
Key Capabilities
- Advanced Pruning Research: Serves as a research checkpoint for exploring semi-structured weight sparsity in large language models.
- CWS Method Implementation: Demonstrates the effectiveness of CWS, which greedily selects weights based on marginal cost derived from an evolving inverse-Hessian factor.
- Performance with Sparsity: Achieves a WikiText-2 Perplexity (PPL) of 9.12 at 30% sparsity, compared to the dense model's 8.64, showcasing efficient pruning with minimal performance degradation.
- Semi-Structured Sparsity: Achieves 30% sparsity by removing the same fraction of weights from every output row, making it suitable for hardware acceleration with sparse kernels.
Good for
- Researchers: Ideal for those studying LLM compression, pruning techniques, and efficient inference.
- Understanding CWS: Provides a practical example of the CWS method and its application to a LLaMA-based model.
- Benchmarking Pruning Methods: Can be used as a baseline or comparison point for evaluating new pruning algorithms.