ikimyaii/llama-7b-cws-50pct
The ikimyaii/llama-7b-cws-50pct model is a 7 billion parameter language model based on OpenLLaMA-7B, developed by ikimyaii. It has been pruned to 50% weight sparsity using the Correlation-aware Weight Sparsity (CWS) method, which is a one-shot, post-training technique that extends SparseGPT with a full-Hessian selection criterion. This model is designed for research into efficient LLM inference through structured sparsity, offering a reduced parameter count compared to its dense counterpart while maintaining a 4096-token context length.
Loading preview...
ikimyaii/llama-7b-cws-50pct: A Sparsified OpenLLaMA-7B Model
This model is a 7 billion parameter variant of the openlm-research/open_llama_7b base model, developed by ikimyaii. It has been significantly optimized for efficiency through Correlation-aware Weight Sparsity (CWS), a one-shot, post-training pruning method. The model achieves 50% weight sparsity by removing weights from every output row in a semi-structured manner, specifically targeting nn.Linear layers within the decoder blocks and the lm_head.
Key Capabilities & Characteristics
- 50% Weight Sparsity: Achieved through the CWS method, which extends SparseGPT's Optimal Brain Surgeon (OBS) with a full-Hessian selection criterion to account for cross-weight interactions.
- Research Checkpoint: Primarily intended for studying the effects and performance of pruning techniques in large language models.
- Post-Training Pruning: Sparsity is applied after the initial training of the base model, without further retraining.
- Semi-Structured Pruning: Weights are removed from every output row, with greedy selection confined to 128-column blocks (Block CWS).
Good For
- Sparsity Research: Ideal for researchers investigating post-training pruning methods, particularly CWS, and their impact on LLM performance and efficiency.
- Benchmarking Sparse Models: Useful for comparing the performance of CWS against other pruning techniques like SparseGPT, Wanda, RIA, and AWP, as detailed in the evaluation section.
- Exploring Model Compression: Provides a practical example of how significant weight reduction can be achieved in LLMs, albeit with some performance trade-offs compared to the dense base model (e.g., WikiText-2 PPL of 11.92 vs. 8.64 for dense).
Note: This model is not instruction-tuned or safety-tuned and inherits the biases of its base model. The weights are stored densely with zeros, meaning sparsity alone does not provide a speedup without specialized sparse kernels or hardware.