ikimyaii/llama-7b-ria-70pct
The ikimyaii/llama-7b-ria-70pct model is a 7 billion parameter LLaMA-based language model, specifically an OpenLLaMA-7B variant, that has been pruned to 70% weight sparsity using the RIA method. This research checkpoint is designed for studying pruning techniques and weight sparsity in large language models, rather than for direct application in text generation. It demonstrates the effects of high sparsity on model performance, serving as a baseline for comparison in sparsity research.
Loading preview...
Overview
This model, ikimyaii/llama-7b-ria-70pct, is a 7 billion parameter OpenLLaMA-7B variant that has undergone significant pruning. It achieves approximately 70% weight sparsity using the RIA method (Relative Importance against both row and column, scaled by activation size), as detailed in Zhang et al., ICLR 2024. The pruning was applied one-shot to nn.Linear layers across 32 decoder layers and the lm_head, totaling 225 layers, while embeddings and norms remain dense.
Key Characteristics
- Base Model:
openlm-research/open_llama_7b - Pruning Method: RIA (Relative Importance against both row and column, scaled by activation size)
- Sparsity: Achieved 69.99% weight sparsity, with zeros stored at pruned positions.
- Research Focus: Primarily a research checkpoint for studying pruning techniques and their impact on LLMs.
- No Retraining/Weight Correction: Pruning was performed without subsequent retraining or weight correction.
Limitations
- Research-Oriented: Not instruction-tuned or safety-tuned; inherits biases from the base model.
- Performance Impact: At 70-80% sparsity, the model does not produce usable text, and zero-shot accuracy is near chance, as indicated by a WikiText-2 PPL of 104.14 (compared to 8.64 for the dense model).
- No Immediate Speedup: Weights are stored densely, meaning sparsity alone does not provide speedup without specialized sparse kernels or hardware.