ikimyaii/llama-7b-awp-70pct

TEXT GENERATIONPricing:Input $0.4 / Cached $0.08 / Output $0.8Concurrent Unit Cost:1Model Size:7BQuant:FP8Context Size:4kPublished:May 11, 2026License:mitArchitecture:Transformer Open Weights Featherless Exclusive Cold

ikimyaii/llama-7b-awp-70pct is a 7 billion parameter LLaMA-based model, specifically OpenLLaMA-7B, that has been pruned to 70% weight sparsity using the AWP method. This model is a research checkpoint designed for studying pruning techniques, achieving a WikiText-2 PPL of 39.22, significantly outperforming other one-shot pruning methods like SparseGPT and Wanda at the same sparsity level. Its primary use case is for research into weight sparsity and its impact on language model performance, rather than for generating usable text.

Loading preview...

Overview

ikimyaii/llama-7b-awp-70pct is a 7 billion parameter model derived from openlm-research/open_llama_7b. It has been pruned one-shot to 70% weight sparsity using the AWP (Activation-Weighted Pruning) method. This model serves as a baseline in the CWS comparison project, focusing on the effectiveness of different pruning techniques.

Key Capabilities & Characteristics

  • High Sparsity: Achieves 70% weight sparsity across nn.Linear layers in all 32 decoder layers and lm_head, totaling 225 pruned layers. Embeddings and norms remain dense.
  • AWP Method: Utilizes the AWP method (Liu et al., 2025, arXiv:2506.10205), which minimizes activation-weighted reconstruction error through iterative hard thresholding.
  • Superior Perplexity: Demonstrates significantly better performance compared to other one-shot pruning methods at 70% sparsity. On WikiText-2, this AWP-pruned model achieved a PPL of 39.22, outperforming SparseGPT (71.20), Wanda (82.10), and RIA (104.14) (dense PPL 8.64).
  • Research Checkpoint: This model is specifically a research artifact for studying pruning. It is not instruction-tuned or safety-tuned.

Good For

  • Pruning Research: Ideal for researchers investigating the impact and effectiveness of weight sparsity techniques, particularly the AWP method.
  • Comparative Studies: Useful as a baseline for comparing new pruning algorithms against established one-shot methods.
  • Understanding Sparsity: Provides a practical example of a highly sparse LLM for analysis of its internal structure and performance characteristics.

Limitations

  • Not for Text Generation: Due to its high sparsity, the model does not produce usable text and zero-shot accuracy is near chance.
  • No Speedup: Weights are stored densely with zeros; sparsity alone does not provide inference speedup without specialized sparse kernels or hardware.

Finetunes
1 models