ikimyaii/llama-7b-wanda-50pct

TEXT GENERATIONPricing:Input $0.4 / Cached $0.08 / Output $0.8Concurrent Unit Cost:1Model Size:7BQuant:FP8Context Size:4kPublished:May 11, 2026License:mitArchitecture:Transformer Open Weights Featherless Exclusive Cold

The ikimyaii/llama-7b-wanda-50pct model is a 7 billion parameter language model based on OpenLLaMA-7B, pruned to 50% weight sparsity using the Wanda method. This one-shot pruning technique removes the lowest-scoring weights in each output row, resulting in a semi-structured sparse model. It serves as a research checkpoint for studying pruning techniques and is primarily used for evaluating the impact of sparsity on model performance, particularly in comparison to dense models and other pruning methods.

Loading preview...

ikimyaii/llama-7b-wanda-50pct: A Sparse LLaMA-7B Model

This model is a 7 billion parameter variant of the OpenLLaMA-7B base model, meticulously pruned to 50% weight sparsity using the Wanda method. Wanda is a one-shot pruning technique that scores weights based on their magnitude and input-channel activation RMS, removing the lowest-scoring weights in each output row. This results in a semi-structured sparse model where 50% of the weights in nn.Linear layers (attention q/k/v/o, MLP gate/up/down, and lm_head) are set to zero, while embeddings and norms remain dense.

Key Capabilities

  • Research Checkpoint: Primarily designed for studying the effects of weight pruning on large language models.
  • Wanda Pruning: Demonstrates the application of the Wanda method for achieving high sparsity.
  • Performance Baseline: Serves as a comparison point against dense models and other pruning techniques like CWS, SparseGPT, RIA, and AWP.

Good For

  • Sparsity Research: Ideal for researchers investigating the trade-offs between model sparsity and performance.
  • Benchmarking Pruning Methods: Useful for comparing the effectiveness of different pruning algorithms on LLaMA-7B architecture.
  • Understanding Model Compression: Provides insights into how weight sparsity impacts perplexity and common sense reasoning tasks, despite a performance drop compared to its dense counterpart (e.g., WikiText-2 PPL of 11.43 vs. 8.64 for dense).

It's important to note that this model is a research checkpoint, not instruction-tuned or safety-tuned, and stores weights densely with zeros at pruned positions, meaning sparsity alone does not provide speedup without specialized hardware or kernels.

Finetunes
1 models