ikimyaii/llama-7b-awp-50pct

TEXT GENERATIONPricing:Input $0.4 / Cached $0.08 / Output $0.8Concurrent Unit Cost:1Model Size:7BQuant:FP8Context Size:4kPublished:May 11, 2026License:mitArchitecture:Transformer Open Weights Featherless Exclusive Cold

The ikimyaii/llama-7b-awp-50pct model is a 7 billion parameter LLaMA-based language model that has been pruned to 50% weight sparsity using the Activation-Weighted Pruning (AWP) method. Developed by ikimyaii, this model serves as a research checkpoint for studying the effects of one-shot semi-structured pruning on large language models. It demonstrates the trade-offs between sparsity and performance, achieving a WikiText-2 PPL of 10.62 compared to the dense model's 8.64. This model is particularly useful for researchers exploring efficient model architectures and the impact of pruning techniques on LLM performance.

Loading preview...

ikimyaii/llama-7b-awp-50pct: A Sparse LLaMA-7B Model

This model is a 7 billion parameter variant of the OpenLLaMA-7B base model, specifically engineered with 50% weight sparsity. It was created using the Activation-Weighted Pruning (AWP) method, a one-shot pruning technique that removes 50% of weights in a semi-structured manner from every output row of nn.Linear layers, including attention and MLP components, as well as the lm_head.

Key Characteristics & Performance

  • Pruning Method: AWP (Liu et al., 2025, arXiv:2506.10205), which minimizes activation-weighted reconstruction error.
  • Sparsity: Achieves exactly 50% weight sparsity across 225 layers, while embeddings and norms remain dense.
  • Performance: While introducing sparsity, the model maintains reasonable performance compared to its dense counterpart. For instance, it achieves a WikiText-2 Perplexity (PPL) of 10.62, which is competitive among other 50% sparse methods like CWS (11.92), SparseGPT (12.70), Wanda (11.43), and RIA (11.20).
  • Evaluation: Zero-shot evaluation on lm-evaluation-harness shows an average accuracy of 0.591 across 5 tasks (ARC-Easy, ARC-Challenge, HellaSwag, PIQA, WinoGrande), compared to 0.639 for the dense model.

Intended Use Cases

  • Research on Model Pruning: Ideal for researchers investigating the effectiveness and impact of one-shot pruning techniques like AWP on large language models.
  • Efficiency Studies: Useful for exploring the trade-offs between model size, computational efficiency (when combined with sparse kernels), and performance.
  • Baseline Comparison: Serves as a baseline in the CWS comparison for evaluating new sparsity methods.

Limitations

  • This is a research checkpoint and is not instruction-tuned or safety-tuned.
  • Weights are stored densely; sparsity alone does not provide speedup without specialized sparse kernels or hardware.

Finetunes
1 models