ikimyaii/llama-7b-ria-80pct

TEXT GENERATIONPricing:Input $0.4 / Cached $0.08 / Output $0.8Concurrent Unit Cost:1Model Size:7BQuant:FP8Context Size:4kPublished:May 11, 2026License:mitArchitecture:Transformer Open Weights Featherless Exclusive Cold

The ikimyaii/llama-7b-ria-80pct is a 7 billion parameter LLaMA-based language model, derived from OpenLLaMA-7B, that has been pruned to 80% weight sparsity using the RIA (Relative Importance with Activation) method. This model serves as a research checkpoint for studying one-shot pruning techniques, specifically as a baseline in the CWS comparison. It is characterized by its high sparsity level, with 79.98% of its weights set to zero, and is primarily intended for academic evaluation of pruning methods rather than practical text generation due to significant performance degradation.

Loading preview...

Overview

ikimyaii/llama-7b-ria-80pct is a 7 billion parameter language model based on openlm-research/open_llama_7b. It has been one-shot pruned to 80% weight sparsity using the RIA (Relative Importance with Activation) method, where the same fraction of weights is removed from every output row (semi-structured pruning). This model is a research checkpoint, serving as a baseline in the CWS comparison for weight sparsity studies.

Key Characteristics

  • Base Model: Derived from openlm-research/open_llama_7b.
  • Pruning Method: Employs RIA (Zhang et al., ICLR 2024), which scores weights based on their relative importance against both their row and column, scaled by activation size.
  • Sparsity Level: Achieves 79.98% sparsity, with 225 nn.Linear layers (attention q/k/v/o and MLP gate/up/down) and lm_head pruned. Embeddings and norms remain dense.
  • No Retraining: The model undergoes no retraining or weight correction after pruning.

Good for

  • Research on Pruning Techniques: Ideal for academic study and comparison of one-shot weight sparsity methods, particularly within the context of the CWS project.
  • Understanding Sparsity Impact: Useful for analyzing the effects of high sparsity levels (80%) on model performance without retraining.

Limitations

  • Not for Practical Use: Due to its high sparsity, the model's zero-shot accuracy is near chance, and it does not produce usable text. It is not instruction-tuned or safety-tuned.
  • No Speedup: Weights are stored densely with zeros; sparsity alone does not provide speedup without specialized sparse kernels or event-driven hardware.

Finetunes
1 models