ikimyaii/llama-7b-ria-30pct

TEXT GENERATIONPricing:Input $0.4 / Cached $0.08 / Output $0.8Concurrent Unit Cost:1Model Size:7BQuant:FP8Context Size:4kPublished:May 11, 2026License:mitArchitecture:Transformer Open Weights Featherless Exclusive Cold

ikimyaii/llama-7b-ria-30pct is a 7 billion parameter language model based on OpenLLaMA-7B, pruned to 30% weight sparsity using the RIA method. This model is a research checkpoint for studying one-shot pruning techniques, achieving a WikiText-2 PPL of 8.96, which is competitive among various sparsity methods. It is primarily intended for research into efficient model architectures and the effects of structured sparsity.

Loading preview...

Overview

ikimyaii/llama-7b-ria-30pct is a 7 billion parameter language model derived from the openlm-research/open_llama_7b base model. It has been pruned one-shot to approximately 30% weight sparsity using the RIA (Relative Importance Averaging) method, as described by Zhang et al. (ICLR 2024). This model serves as a baseline in the CWS comparison project, focusing on the study of efficient model architectures through pruning.

Key Characteristics

  • Base Model: OpenLLaMA-7B.
  • Pruning Method: RIA, which scores weights based on their relative importance within both their row and column, scaled by activation size.
  • Sparsity: Achieves 29.98% weight sparsity, with the same fraction removed from every output row (semi-structured).
  • Pruned Components: All nn.Linear layers within the 32 decoder layers (attention q/k/v/o and MLP gate/up/down) plus lm_head are pruned, totaling 225 layers. Embeddings and norms remain dense.
  • No Retraining: The model is pruned one-shot without any subsequent retraining or weight correction.

Evaluation and Performance

On the WikiText-2 perplexity (PPL) benchmark, this RIA-pruned model achieved 8.96 PPL (dense baseline 8.64). This performance is competitive with other one-shot pruning methods at 30% sparsity, such as Wanda (9.04 PPL) and SparseGPT (9.18 PPL), and is very close to AWP (8.92 PPL).

Good for

  • Research on Model Sparsity: Ideal for researchers investigating the effects of one-shot pruning techniques, particularly the RIA method.
  • Benchmarking Pruning Algorithms: Useful as a baseline for comparing new sparsity methods against established techniques.
  • Exploring Efficient Architectures: Provides a practical example of a significantly sparse LLM for studying memory and computational efficiency implications.

Limitations

  • This is a research checkpoint and is not instruction-tuned or safety-tuned.
  • It inherits biases from its base model.
  • Weights are stored densely with zeros; sparsity alone does not provide speedup without specialized sparse kernels or hardware.

Finetunes
1 models