ikimyaii/llama-7b-awp-60pct
The ikimyaii/llama-7b-awp-60pct model is a 7 billion parameter language model based on OpenLLaMA-7B, developed by ikimyaii. It features 60% weight sparsity achieved through a one-shot pruning method called AWP (Activation-Weighted Pruning), which minimizes activation-weighted reconstruction error. This model is a research checkpoint for studying pruning techniques, demonstrating significantly improved perplexity (14.69 PPL on WikiText-2) compared to other 60% sparse methods like SparseGPT and Wanda, making it suitable for research into efficient LLM architectures.
Loading preview...
Overview
The ikimyaii/llama-7b-awp-60pct model is a 7 billion parameter language model derived from openlm-research/open_llama_7b. Its key differentiator is its 60% weight sparsity, achieved using the AWP (Activation-Weighted Pruning) method. This model serves as a research checkpoint for evaluating pruning techniques, specifically within the context of the CWS comparison framework.
Key Capabilities
- High Sparsity: Achieves 60% weight sparsity across
nn.Linearlayers in the 32 decoder layers andlm_head, totaling 225 pruned layers. - Efficient Pruning Method: Utilizes AWP, a one-shot pruning technique that minimizes activation-weighted reconstruction error, outperforming other methods like SparseGPT and Wanda in perplexity.
- Improved Perplexity: Demonstrates a WikiText-2 PPL of 14.69 at 60% sparsity, significantly better than CWS (18.58), SparseGPT (20.76), Wanda (18.11), and RIA (18.54) at the same sparsity level (dense PPL 8.64).
Good For
- Research into LLM Pruning: Ideal for researchers studying the effectiveness and impact of different weight pruning techniques on large language models.
- Benchmarking Sparse Models: Can be used as a baseline or comparison point for new sparsity methods, particularly those aiming for high sparsity levels with minimal performance degradation.
- Exploring Model Efficiency: Useful for understanding the trade-offs between model size, sparsity, and performance, especially in scenarios where computational resources are constrained.
Limitations: This model is a research checkpoint and is not instruction-tuned or safety-tuned. Its weights are stored densely, meaning sparsity alone does not provide a speedup without specialized sparse kernels or hardware.