ikimyaii/llama-7b-awp-40pct

TEXT GENERATIONPricing:Input $0.4 / Cached $0.08 / Output $0.8Concurrent Unit Cost:1Model Size:7BQuant:FP8Context Size:4kPublished:May 11, 2026License:mitArchitecture:Transformer Open Weights Featherless Exclusive Cold

The ikimyaii/llama-7b-awp-40pct model is a 7 billion parameter LLaMA-based language model, derived from OpenLLaMA-7B. It has been pruned to 40% weight sparsity using the AWP method, achieving a WikiText-2 perplexity of 9.44, which is competitive among sparse models. This model serves as a research checkpoint for studying pruning techniques and their impact on language model performance.

Loading preview...

Model Overview

This model, ikimyaii/llama-7b-awp-40pct, is a 7 billion parameter variant of the OpenLLaMA-7B base model. It has undergone significant pruning, achieving 40% weight sparsity using the AWP (Activation-Weighted Pruning) method. AWP minimizes the activation-weighted reconstruction error through iterative hard thresholding, initialized from a Wanda mask.

Key Characteristics & Performance

  • Base Model: openlm-research/open_llama_7b
  • Sparsity Method: AWP (Liu et al., 2025, arXiv:2506.10205)
  • Achieved Sparsity: 40.01% across 225 layers, including nn.Linear layers in the 32 decoder layers and lm_head.
  • Perplexity (WikiText-2): Achieved a perplexity of 9.44, outperforming other sparsity methods like Wanda (9.76), SparseGPT (10.24), and RIA (9.54) at the same 40% sparsity level (dense PPL 8.64).
  • Research Focus: This model is primarily a research checkpoint for investigating pruning techniques and their effectiveness.

Limitations

  • Research-Oriented: These checkpoints are for studying pruning and are not instruction-tuned or safety-tuned.
  • Inherited Biases: It inherits the biases present in its base model.
  • No Immediate Speedup: Weights are stored densely with zeros; sparsity alone does not provide speedup without specialized sparse kernels or event-driven hardware.

Finetunes
1 models