ikimyaii/llama-7b-sparsegpt-60pct
The ikimyaii/llama-7b-sparsegpt-60pct model is a 7 billion parameter LLaMA-based causal language model, specifically a pruned version of OpenLLaMA-7B. It was created by ikimyaii using the SparseGPT method to achieve 60% weight sparsity, meaning 60% of its weights are zeroed out. This model serves as a research checkpoint for studying neural network pruning techniques, particularly for understanding the effects of sparsity on large language models.
Loading preview...
Overview
The ikimyaii/llama-7b-sparsegpt-60pct model is a 7 billion parameter variant of the OpenLLaMA-7B base model, distinguished by its significant weight sparsity. Developed by ikimyaii, this model has been pruned to approximately 60% weight sparsity using the SparseGPT method. This process involves setting 60% of the model's weights to zero, specifically within the nn.Linear layers of the 32 decoder layers and the lm_head, while embeddings and norms remain dense. The pruning is performed "one-shot" without subsequent retraining, making it a valuable research checkpoint for analyzing the impact of sparsity on LLMs.
Key Capabilities
- High Weight Sparsity: Achieves 59.99% weight sparsity using the SparseGPT method, a technique that scores weights by OBS saliency and applies Optimal Brain Surgeon corrections.
- Research Baseline: Serves as a baseline model in the CWS comparison, facilitating studies on various pruning techniques.
- Pruning Analysis: Provides a practical example for understanding how significant weight reduction affects model performance and characteristics without retraining.
Good for
- Neural Network Pruning Research: Ideal for researchers investigating the effects of weight sparsity on large language models and comparing different pruning methodologies.
- Understanding SparseGPT: Useful for those looking to study the SparseGPT method's application and its outcomes on a LLaMA-based architecture.
- Exploring Model Efficiency: Can be used to explore the trade-offs between model size (due to sparsity) and performance, particularly in scenarios where sparse hardware or kernels might be utilized.
Limitations: This model is a research checkpoint and is not instruction-tuned or safety-tuned. Its weights are stored densely, meaning sparsity alone does not provide speedup without specialized sparse kernels or event-driven hardware.