ikimyaii/llama-7b-sparsegpt-80pct

TEXT GENERATIONPricing:Input $0.4 / Cached $0.08 / Output $0.8Concurrent Unit Cost:1Model Size:7BQuant:FP8Context Size:4kPublished:May 11, 2026License:mitArchitecture:Transformer Open Weights Featherless Exclusive Cold

The ikimyaii/llama-7b-sparsegpt-80pct is a 7 billion parameter LLaMA-based model, specifically an OpenLLaMA-7B variant, that has been pruned to 80% weight sparsity using the SparseGPT method. This model serves as a research checkpoint for studying pruning techniques, particularly within the context of the CWS comparison. Its primary differentiator is its high sparsity level achieved without retraining, making it valuable for research into efficient model compression and the effects of extreme sparsity on performance.

Loading preview...

Overview

This model, ikimyaii/llama-7b-sparsegpt-80pct, is a 7 billion parameter variant of the OpenLLaMA-7B base model that has undergone significant pruning. It achieves approximately 80% weight sparsity using the SparseGPT one-shot pruning method, where the same fraction of weights is removed from every output row (semi-structured). This model is a baseline in the CWS comparison project, focusing on the study of weight sparsity in large language models.

Key Characteristics

  • Base Model: OpenLLaMA-7B.
  • Pruning Method: SparseGPT, a one-shot technique that scores weights by OBS saliency and applies column-ordered Optimal Brain Surgeon corrections.
  • Sparsity Level: Achieves 79.98% sparsity, with pruning applied to nn.Linear layers in the 32 decoder layers and lm_head (225 layers total).
  • No Retraining: The model was pruned without any subsequent retraining, making it a direct study of the impact of pruning.
  • Research Focus: Primarily intended as a research checkpoint for understanding pruning effects and comparing different sparsity methods.

Limitations and Intended Use

  • Research Only: These are research checkpoints for studying pruning; they are not instruction-tuned or safety-tuned and inherit the base model's biases.
  • Performance Impact: At 70-80% sparsity, the model's zero-shot accuracy is near chance, and it does not produce usable text for general applications. For example, its WikiText-2 PPL is 1,103.6 compared to 8.64 for the dense model.
  • No Immediate Speedup: Weights are stored densely with zeros; sparsity alone does not provide speedup without specialized sparse kernels or hardware.

This model is ideal for researchers investigating the boundaries of model compression, the effectiveness of SparseGPT, and the trade-offs between sparsity and performance in LLMs.

Finetunes
1 models