ikimyaii/llama-7b-awp-30pct
The ikimyaii/llama-7b-awp-30pct model is a 7 billion parameter language model based on OpenLLaMA-7B, which has been pruned to 30% weight sparsity using the Activation-Weighted Pruning (AWP) method. This semi-structured pruning technique removes the same fraction of weights from every output row in key linear layers, resulting in a more efficient model. It serves as a research checkpoint for studying pruning techniques and demonstrates improved perplexity on WikiText-2 compared to other sparsity methods like SparseGPT and Wanda, making it suitable for research into efficient model deployment.
Loading preview...
Overview
This model, ikimyaii/llama-7b-awp-30pct, is a 7 billion parameter variant of the openlm-research/open_llama_7b base model. It has been significantly optimized through Activation-Weighted Pruning (AWP), achieving a 30% weight sparsity. This pruning was performed one-shot, meaning it was applied directly without iterative retraining, and targets nn.Linear layers across the 32 decoder layers and the lm_head.
Key Capabilities
- High Sparsity: Achieves 30% weight sparsity, reducing the number of active parameters.
- AWP Method: Utilizes the AWP method, which minimizes activation-weighted reconstruction error, leading to effective pruning.
- Improved Perplexity: Demonstrates superior performance in terms of WikiText-2 PPL (8.92) compared to other 30% sparse methods like CWS (9.12), SparseGPT (9.18), and Wanda (9.04), indicating better language modeling capability at high sparsity.
Good For
- Pruning Research: Ideal as a research checkpoint for studying and comparing different model pruning techniques.
- Efficient Model Exploration: Useful for developers and researchers interested in deploying more efficient, sparse large language models.
- Baseline Comparisons: Serves as a strong baseline in projects like CWS for evaluating new sparsity methods.
Limitations
It's important to note that this model is a research checkpoint and is not instruction-tuned or safety-tuned. It inherits the biases of its base model. Additionally, the weights are stored densely with zeros, meaning sparsity alone does not provide speedup without specialized sparse kernels or event-driven hardware.