ikimyaii/llama-7b-wanda-80pct
The ikimyaii/llama-7b-wanda-80pct model is a 7 billion parameter LLaMA-based language model, specifically an OpenLLaMA-7B variant, that has been pruned to 80% weight sparsity using the Wanda method. Developed by ikimyaii as a research checkpoint for studying pruning techniques, it features semi-structured sparsity where the same fraction of weights is removed from every output row. This model serves as a baseline for comparison in sparse LLM research, demonstrating the impact of one-shot pruning on model performance without retraining.
Loading preview...
Overview
ikimyaii/llama-7b-wanda-80pct is a research checkpoint derived from the OpenLLaMA-7B model, pruned to 80% weight sparsity using the Wanda method. This model is a baseline for the CWS comparison, designed to study the effects of one-shot pruning on large language models. The pruning process removes weights from every nn.Linear layer (225 layers in total), while embeddings and norms remain dense. The Wanda method scores weights by their magnitude multiplied by the input-channel activation RMS, removing the lowest-scoring ones without weight correction or retraining.
Key Characteristics
- Base Model: OpenLLaMA-7B
- Pruning Method: Wanda (Sun et al., ICLR 2024, arXiv:2306.11695)
- Sparsity: Achieves 79.98% weight sparsity, with weights stored densely (zeros at pruned positions).
- No Retraining: Pruning is applied one-shot without any subsequent fine-tuning.
- Research Focus: Primarily intended for academic study of pruning techniques and their impact on model performance.
Limitations and Use Cases
This model is a research checkpoint and has significant limitations for general use. It is not instruction-tuned or safety-tuned and inherits biases from its base model. Due to the high sparsity, its zero-shot accuracy is near chance, and it does not produce usable text. Furthermore, the dense storage of weights means sparsity alone does not provide speedup without specialized sparse kernels or hardware. It is suitable for researchers investigating pruning methods and comparing different sparsity techniques, particularly within the context of the Sparse-LLMs-on-Loihi project.