ikimyaii/llama-7b-ria-40pct

TEXT GENERATIONPricing:Input $0.4 / Cached $0.08 / Output $0.8Concurrent Unit Cost:1Model Size:7BQuant:FP8Context Size:4kPublished:May 11, 2026License:mitArchitecture:Transformer Open Weights Featherless Exclusive Cold

ikimyaii/llama-7b-ria-40pct is a 7 billion parameter language model based on OpenLLaMA-7B, pruned to 40% weight sparsity using the RIA method. This model is a research checkpoint designed for studying pruning techniques, specifically demonstrating semi-structured sparsity where the same fraction of weights is removed from every output row. It achieves a WikiText-2 PPL of 9.54, making it a strong baseline for evaluating sparse LLMs. Its primary use case is for research in model compression and efficient AI.

Loading preview...

Overview

ikimyaii/llama-7b-ria-40pct is a 7 billion parameter language model derived from the openlm-research/open_llama_7b base model. It has been pruned to 40% weight sparsity using the RIA method (Relative Importance against both its row and its column, scaled by activation size), as described by Zhang et al. (ICLR 2024). This model serves as a research checkpoint for studying the effects of one-shot pruning without retraining.

Key Capabilities

  • High Sparsity: Achieves 39.99% weight sparsity across 225 nn.Linear layers, including attention and MLP components, while keeping embeddings and norms dense.
  • Efficient Pruning Method: Utilizes the RIA method, which scores weights based on their relative importance within rows and columns, scaled by activation size.
  • Competitive Performance: Demonstrates a WikiText-2 Perplexity (PPL) of 9.54 at 40% sparsity, outperforming other methods like CWS, SparseGPT, and Wanda in direct comparisons (dense PPL 8.64).

Good for

  • Research in Model Pruning: Ideal for researchers investigating weight sparsity, pruning techniques, and the performance of sparse LLMs.
  • Benchmarking: Can be used as a baseline for comparing new pruning methods against established techniques like RIA.
  • Understanding Sparsity Effects: Provides a practical example of a significantly sparse model for analyzing the impact of weight removal on language model performance.

Limitations

  • This model is a research checkpoint and is not instruction-tuned or safety-tuned.
  • Weights are stored densely with zeros, meaning sparsity alone does not provide speedup without specialized sparse kernels or hardware.

Finetunes
1 models