ikimyaii/llama-7b-sparsegpt-40pct
The ikimyaii/llama-7b-sparsegpt-40pct model is a 7 billion parameter LLaMA-based language model, derived from OpenLLaMA-7B, that has been pruned to 40% weight sparsity using the SparseGPT method. Developed by ikimyaii, this research checkpoint focuses on exploring the effects of one-shot weight pruning without retraining. It serves as a baseline for sparsity studies, demonstrating the impact of pruning on perplexity while maintaining a compact model size.
Loading preview...
Model Overview
This model, ikimyaii/llama-7b-sparsegpt-40pct, is a 7 billion parameter LLaMA-based language model derived from openlm-research/open_llama_7b. It has been one-shot pruned to 40% weight sparsity using the SparseGPT method, with a semi-structured approach where the same fraction is removed from every output row. This model is primarily a research checkpoint designed for studying the effects of pruning and serves as a baseline in the CWS comparison.
Key Characteristics
- Base Model:
openlm-research/open_llama_7b - Pruning Method: SparseGPT (Frantar & Alistarh, 2023, arXiv:2301.00774)
- Sparsity: Achieved 39.99% weight sparsity (target 40%)
- Weight Correction: Optimal Brain Surgeon (OBS) second-order correction
- Retraining: None after pruning
- Pruned Components: All
nn.Linearlayers in the 32 decoder layers (attention q/k/v/o and MLP gate/up/down) pluslm_head, totaling 225 layers. Embeddings and norms remain dense.
Evaluation & Limitations
Evaluated on WikiText-2 PPL, this SparseGPT model achieved 10.24 PPL at 40% sparsity (compared to a dense PPL of 8.64). It is important to note that these are research checkpoints and are not instruction-tuned or safety-tuned. They inherit the biases of the base model. Furthermore, the weights are stored densely with zeros at pruned positions, meaning sparsity alone does not provide a speedup without specialized sparse kernels or event-driven hardware.