ikimyaii/llama-7b-cws-70pct
The ikimyaii/llama-7b-cws-70pct model is a 7 billion parameter LLaMA-based language model, derived from OpenLLaMA-7B, that has been pruned to 70% weight sparsity using the Correlation-aware Weight Sparsity (CWS) method. This semi-structured pruning technique, which extends SparseGPT, focuses on identifying and removing correlated input channels to optimize sparsity while minimizing performance degradation. It is a research checkpoint designed for studying pruning techniques, particularly CWS, and is not intended for general text generation due to its high sparsity level.
Loading preview...
ikimyaii/llama-7b-cws-70pct: A Research Model for Weight Sparsity
This model is a 7 billion parameter variant of OpenLLaMA-7B that has undergone significant pruning to achieve 70% weight sparsity. Developed by ikimyaii, it utilizes the Correlation-aware Weight Sparsity (CWS) method, a novel one-shot, post-training pruning technique. CWS enhances the Optimal Brain Surgeon (OBS) weight correction by incorporating a full-Hessian selection criterion, allowing it to identify and remove "cancellation groups" of correlated weights more effectively than methods relying on isolated per-weight saliency scores.
Key Capabilities & Characteristics
- Advanced Pruning Method: Employs CWS, which considers cross-weight interactions, leading to more effective sparsity compared to methods like SparseGPT, Wanda, RIA, and AWP at similar sparsity levels.
- Semi-Structured Sparsity: Achieves 70% sparsity by removing weights from every output row of
nn.Linearlayers in the decoder, while embeddings and norms remain dense. - Research Checkpoint: Primarily intended for studying the efficacy of the CWS pruning method and its impact on model performance.
- No Retraining: The pruning is applied one-shot without any subsequent retraining, making it a direct evaluation of the pruning technique itself.
Good for
- Research on LLM Pruning: Ideal for researchers investigating advanced weight sparsity techniques, particularly CWS.
- Comparative Analysis: Useful for comparing the effectiveness of different pruning methods on LLaMA-based architectures.
- Understanding Sparsity Impact: Provides a clear example of a highly sparse model (70%) to study the trade-offs between sparsity and model utility.
Note: This model is a research artifact and is not instruction-tuned or safety-tuned. Due to its high sparsity, it does not produce usable text for general applications, and zero-shot accuracy is near chance. Weights are stored densely, meaning sparsity alone does not provide a speedup without specialized sparse kernels or hardware.