ikimyaii/llama-7b-awp-80pct

TEXT GENERATIONPricing:Input $0.4 / Cached $0.08 / Output $0.8Concurrent Unit Cost:1Model Size:7BQuant:FP8Context Size:4kPublished:May 11, 2026License:mitArchitecture:Transformer Open Weights Featherless Exclusive Cold

The ikimyaii/llama-7b-awp-80pct is a 7 billion parameter LLaMA-based language model, derived from openlm-research/open_llama_7b, that has been pruned to 80% weight sparsity using the AWP (Activation-Weighted Pruning) method. This semi-structured pruning approach removes the same fraction of weights from every output row in key linear layers, making it a research checkpoint for studying efficient model compression. While highly sparse, it is not instruction-tuned or safety-tuned and does not produce usable text at this sparsity level, serving primarily as a baseline for sparsity research.

Loading preview...

Overview

The ikimyaii/llama-7b-awp-80pct model is a 7 billion parameter variant of the openlm-research/open_llama_7b base model, specifically designed as a research checkpoint for studying weight sparsity. It has been pruned one-shot to 80% weight sparsity using the AWP (Activation-Weighted Pruning) method, which minimizes the activation-weighted reconstruction error. This model is a baseline in the CWS comparison project.

Key Characteristics

  • Base Model: Derived from openlm-research/open_llama_7b.
  • Pruning Method: AWP (Liu et al., 2025, arXiv:2506.10205), applied one-shot without retraining.
  • Sparsity: Achieves 80.01% sparsity, with pruning applied to all nn.Linear layers in the 32 decoder layers and lm_head.
  • Research Focus: Primarily intended for research into model compression and sparse neural networks.

Limitations and Use Cases

This model is a research checkpoint and has significant limitations for general use:

  • Not Usable for Text Generation: At 70-80% sparsity, the model does not produce usable text, and zero-shot accuracy is near chance (e.g., WikiText-2 PPL of 230.10 compared to 8.64 for dense).
  • No Instruction or Safety Tuning: It inherits biases from the base model and is not fine-tuned for specific tasks or safety.
  • No Speedup: Weights are stored densely with zeros; sparsity alone does not provide inference speedup without specialized sparse kernels or hardware.

Therefore, this model is good for:

  • Academic Research: Studying the effects of extreme weight sparsity and evaluating pruning methods like AWP.
  • Benchmarking: Serving as a baseline for comparison against other sparsity techniques within the CWS project and related work.

Finetunes
1 models