ikimyaii/llama-7b-wanda-40pct

TEXT GENERATIONPricing:Input $0.4 / Cached $0.08 / Output $0.8Concurrent Unit Cost:1Model Size:7BQuant:FP8Context Size:4kPublished:May 11, 2026License:mitArchitecture:Transformer Open Weights Featherless Exclusive Cold

ikimyaii/llama-7b-wanda-40pct is a 7 billion parameter LLaMA-based model, specifically OpenLLaMA-7B, that has been pruned to 40% weight sparsity using the Wanda method. This one-shot pruning technique removes the lowest-scoring weights based on magnitude and input activation RMS, without retraining or weight correction. It serves as a research checkpoint for studying pruning techniques and offers a WikiText-2 PPL of 9.76, making it suitable for research into efficient model architectures.

Loading preview...

Overview

ikimyaii/llama-7b-wanda-40pct is a 7 billion parameter language model derived from openlm-research/open_llama_7b. This model has been pruned to achieve 40% weight sparsity using the Wanda method, a one-shot pruning technique. It is designed as a research checkpoint for the CWS comparison project, focusing on the study of efficient sparse model architectures.

Key Capabilities

  • 40% Weight Sparsity: Achieved through the Wanda method, which scores weights by magnitude multiplied by input-channel activation RMS and removes the lowest-scoring ones.
  • Research Baseline: Serves as a comparative baseline for various pruning methods, particularly within the CWS project.
  • Efficient Architecture Study: Provides a practical example of a significantly pruned LLM for research into model compression and efficiency.

Evaluation Highlights

On the WikiText-2 perplexity (PPL) benchmark, this model achieved a PPL of 9.76 at 40% sparsity, compared to the dense model's 8.64. This places its performance competitively among other pruning methods like SparseGPT (10.24) and CWS (9.97) at the same sparsity level.

Good for

  • Sparsity Research: Ideal for researchers investigating the impact and effectiveness of one-shot pruning techniques like Wanda on large language models.
  • Model Compression Studies: Useful for experiments on reducing model size and computational requirements without extensive retraining.
  • Comparative Analysis: Can be used as a benchmark model when evaluating new pruning algorithms or sparse inference techniques.

Finetunes
1 models