ikimyaii/llama-7b-sparsegpt-30pct
ikimyaii/llama-7b-sparsegpt-30pct is a 7 billion parameter language model based on OpenLLaMA-7B, developed by ikimyaii. This model has been pruned to 30% weight sparsity using the SparseGPT method, resulting in a semi-structured sparse architecture. It serves as a research checkpoint for studying pruning techniques, offering insights into the trade-offs between sparsity and performance. Its primary use case is for research and comparison within the context of sparse large language models.
Loading preview...
Overview
ikimyaii/llama-7b-sparsegpt-30pct is a research checkpoint derived from the OpenLLaMA-7B base model, pruned to 30% weight sparsity using the SparseGPT method. This model is a baseline in the CWS comparison, focusing on the study of efficient sparse language models. The pruning process was applied one-shot, removing 30% of weights from every output row in nn.Linear layers across the 32 decoder layers and the lm_head, totaling 225 pruned layers.
Key Capabilities
- SparseGPT Pruning: Utilizes the SparseGPT method (Frantar & Alistarh, 2023) for efficient one-shot weight pruning, scoring weights by OBS saliency and applying Optimal Brain Surgeon corrections.
- Research Checkpoint: Primarily intended for research into model sparsity and its impact on performance, serving as a comparative baseline.
- Weight Sparsity: Achieves approximately 29.98% weight sparsity, demonstrating a semi-structured sparse architecture.
Good for
- Sparsity Research: Ideal for researchers studying pruning techniques, sparse model architectures, and their implications for efficiency and performance.
- Comparative Analysis: Useful for comparing different sparsity methods, as demonstrated by its inclusion in the CWS comparison project.
- Understanding Pruning Effects: Provides a practical example of a pruned model to analyze the effects of weight sparsity on language model behavior and perplexity, with a WikiText-2 PPL of 9.18 at 30% sparsity (dense PPL 8.64).
Limitations
It is important to note that this model is a research checkpoint and is not instruction-tuned or safety-tuned. It inherits the biases of its base model. Additionally, the weights are stored densely with zeros at pruned positions, meaning sparsity alone does not provide speedup without specialized sparse kernels or event-driven hardware.