ikimyaii/llama-7b-ria-50pct

TEXT GENERATIONPricing:Input $0.4 / Cached $0.08 / Output $0.8Concurrent Unit Cost:1Model Size:7BQuant:FP8Context Size:4kPublished:May 11, 2026License:mitArchitecture:Transformer Open Weights Featherless Exclusive Cold

The ikimyaii/llama-7b-ria-50pct is a 7 billion parameter LLaMA-based causal language model, derived from OpenLLaMA-7B, that has been pruned to 50% weight sparsity using the RIA (Relative Importance with Activations) method. This model is a research checkpoint designed for studying pruning techniques, specifically demonstrating semi-structured sparsity where 50% of weights are zeroed out. It serves as a baseline in the CWS comparison for sparse LLMs, offering insights into the trade-offs between model size reduction and performance, with a context length of 4096 tokens.

Loading preview...

Overview

ikimyaii/llama-7b-ria-50pct is a 7 billion parameter language model based on the OpenLLaMA-7B architecture. This model has undergone significant pruning, achieving 50% weight sparsity using the RIA (Relative Importance with Activations) method. RIA is a one-shot pruning technique that scores weights based on their relative importance within both their row and column, scaled by activation size. This results in a semi-structured sparsity where half of the weights are zeroed out across specific layers, including nn.Linear layers in the 32 decoder layers and lm_head.

Key Characteristics

  • 50% Weight Sparsity: Achieved through the RIA method, making it a significantly smaller model in terms of non-zero parameters compared to its dense counterpart.
  • Research Checkpoint: Primarily intended for studying pruning techniques and their impact on LLM performance, serving as a baseline in the CWS comparison.
  • No Retraining: The model was pruned one-shot without any subsequent retraining or weight correction, highlighting the effectiveness of the RIA method in maintaining performance post-pruning.
  • Performance Trade-offs: While achieving high sparsity, the model exhibits a performance drop compared to the dense OpenLLaMA-7B, as shown by evaluation metrics like WikiText-2 PPL (11.20 vs. 8.64 for dense) and various zero-shot benchmarks (e.g., ARC-Easy 0.601 vs. 0.678 for dense).

Good for

  • Sparsity Research: Ideal for researchers and developers interested in exploring and comparing different pruning methods for large language models.
  • Understanding Pruning Impact: Provides a clear example of a 50% sparse model without retraining, allowing for direct analysis of how sparsity affects model performance.
  • Baseline Comparisons: Useful as a reference point for evaluating new sparsity techniques against a well-documented RIA-pruned model.

Finetunes
1 models