ikimyaii/llama-7b-ria-60pct

TEXT GENERATIONPricing:Input $0.4 / Cached $0.08 / Output $0.8Concurrent Unit Cost:1Model Size:7BQuant:FP8Context Size:4kPublished:May 11, 2026License:mitArchitecture:Transformer Open Weights Featherless Exclusive Cold

The ikimyaii/llama-7b-ria-60pct model is a 7 billion parameter LLaMA-based language model, derived from OpenLLaMA-7B, that has been pruned to 60% weight sparsity using the RIA (Relative Importance against both row and column) method. This research checkpoint is designed for studying one-shot weight pruning techniques, offering insights into the trade-offs between sparsity and performance. It serves as a baseline in the CWS comparison for sparse LLMs, demonstrating a WikiText-2 PPL of 18.54 at 60% sparsity.

Loading preview...

Overview

The ikimyaii/llama-7b-ria-60pct model is a 7 billion parameter LLaMA-based language model derived from openlm-research/open_llama_7b. It has been one-shot pruned to 60% weight sparsity using the RIA (Relative Importance against both row and column) method, with the same fraction removed from every output row (semi-structured). This model is a research checkpoint primarily intended for studying weight pruning techniques and serves as a baseline in the CWS comparison.

Key Characteristics

  • Base Model: openlm-research/open_llama_7b
  • Pruning Method: RIA (Zhang et al., ICLR 2024), scoring weights by (|W[r,c]| / ‖W[r,:]‖₁ + |W[r,c]| / ‖W[:,c]‖₁) · act_rms[c]^α with α = 0.5.
  • Sparsity: Achieves 59.99% sparsity, targeting 60%.
  • No Retraining or Weight Correction: Pruning is applied one-shot without subsequent fine-tuning.
  • Pruned Layers: All nn.Linear layers in the 32 decoder layers (attention q/k/v/o and MLP gate/up/down) plus lm_head (225 layers total). Embeddings and norms remain dense.

Performance

On the WikiText-2 perplexity (PPL) benchmark, this RIA-pruned model achieves a PPL of 18.54 at 60% sparsity, compared to the dense base model's PPL of 8.64. This places its performance competitively among other one-shot pruning methods like Wanda (18.11 PPL) and SparseGPT (20.76 PPL) at the same sparsity level.

Good for

  • Researching Weight Pruning: Ideal for academics and researchers studying the effects and efficacy of one-shot weight sparsity methods like RIA.
  • Benchmarking Sparse LLMs: Useful as a comparative baseline for evaluating new pruning techniques against established methods.
  • Understanding Sparsity Trade-offs: Provides a practical example of how significant sparsity impacts model performance without retraining.

Finetunes
1 models