ikimyaii/llama-7b-sparsegpt-70pct

TEXT GENERATIONPricing:Input $0.4 / Cached $0.08 / Output $0.8Concurrent Unit Cost:1Model Size:7BQuant:FP8Context Size:4kPublished:May 11, 2026License:mitArchitecture:Transformer Open Weights Featherless Exclusive Cold

The ikimyaii/llama-7b-sparsegpt-70pct model is a 7 billion parameter LLaMA-based language model, specifically a pruned version of OpenLLaMA-7B. Developed by ikimyaii, it achieves 70% weight sparsity using the SparseGPT method, with weights removed semi-structurally from every output row. This model serves as a research checkpoint for studying pruning techniques in large language models, demonstrating the impact of high sparsity on perplexity without retraining. Its primary use case is for research into sparse model architectures and their efficiency implications.

Loading preview...

Overview

The ikimyaii/llama-7b-sparsegpt-70pct model is a 7 billion parameter variant of the OpenLLaMA-7B base model, distinguished by its 70% weight sparsity. This sparsity was achieved using the SparseGPT method, which prunes weights one-shot without any subsequent retraining. The pruning targets nn.Linear layers across the 32 decoder layers and the lm_head, totaling 225 layers, while embeddings and norms remain dense.

Key Capabilities & Characteristics

  • High Sparsity: Achieves approximately 70% weight sparsity, making it a significant research checkpoint for efficient model architectures.
  • SparseGPT Method: Utilizes the SparseGPT method (Frantar & Alistarh, 2023) for pruning, which scores weights by OBS saliency and applies column-ordered Optimal Brain Surgeon corrections.
  • Research Focus: Primarily intended for research into pruning techniques and the behavior of highly sparse models, serving as a baseline in the CWS comparison.

Limitations & Considerations

  • Research Checkpoint: This model is not instruction-tuned or safety-tuned and inherits biases from its base model. It is not designed for direct application in production environments requiring usable text generation.
  • No Speedup Without Sparse Kernels: The weights are stored densely with zeros, meaning sparsity alone does not provide speedup without specialized sparse kernels or event-driven hardware.
  • Performance at High Sparsity: At 70-80% sparsity, the model's ability to produce usable text is significantly degraded, with zero-shot accuracy near chance, as indicated by a WikiText-2 PPL of 71.20 (compared to 8.64 for the dense model).

Finetunes
1 models