ikimyaii/llama-7b-sparsegpt-50pct

TEXT GENERATIONPricing:Input $0.4 / Cached $0.08 / Output $0.8Concurrent Unit Cost:1Model Size:7BQuant:FP8Context Size:4kPublished:May 11, 2026License:mitArchitecture:Transformer Open Weights Featherless Exclusive Cold

The ikimyaii/llama-7b-sparsegpt-50pct model is a 7 billion parameter language model derived from OpenLLaMA-7B, created by ikimyaii. It has been pruned to 50% weight sparsity using the SparseGPT method, making it a research checkpoint for studying model compression. This model is characterized by its significant sparsity, with half of its weights set to zero, and is primarily intended for research into efficient neural network architectures rather than direct application. It serves as a baseline in the CWS comparison for sparsity techniques.

Loading preview...

Overview

The ikimyaii/llama-7b-sparsegpt-50pct model is a 7 billion parameter language model based on openlm-research/open_llama_7b. It has been significantly compressed, achieving 50% weight sparsity through a one-shot pruning process using the SparseGPT method. This model is a research checkpoint, specifically designed for investigating the effects and performance of model pruning, and serves as a baseline in the CWS comparison.

Key Characteristics

  • High Sparsity: 50% of the model's weights are pruned, specifically within nn.Linear layers across 32 decoder layers and the lm_head.
  • SparseGPT Method: Utilizes the SparseGPT technique, which involves scoring weights by OBS saliency and applying column-ordered Optimal Brain Surgeon corrections.
  • No Retraining: The model was pruned without any subsequent retraining, making it a direct evaluation of one-shot sparsity.
  • Research Focus: Primarily intended for academic and research purposes to study the impact of sparsity on large language models.

Performance Insights

While achieving 50% sparsity, the model exhibits a performance drop compared to its dense counterpart. For instance, its WikiText-2 PPL is 12.70 (vs. 8.64 for dense), and its average accuracy across 5 tasks (ARC-Easy, ARC-Challenge, HellaSwag, PIQA, WinoGrande) is 0.599 (vs. 0.639 for dense). This highlights the trade-off between model size reduction and performance in one-shot pruning without fine-tuning.

Good for

  • Sparsity Research: Ideal for researchers studying model compression, pruning techniques, and the performance implications of high sparsity.
  • Baseline Comparisons: Useful as a comparative baseline for developing and evaluating new sparse model architectures or pruning methods.
  • Understanding Pruning Effects: Provides a clear example of a highly sparse model's behavior without post-pruning fine-tuning.

Finetunes
1 models