cosmos1030/gmp-kd3e-1-s70pct-lr1e-4_20260901_080954

TEXT GENERATIONPricing:Input $0.4 / Cached $0.08 / Output $0.8Concurrent Unit Cost:1Model Size:4BQuant:BF16Context Size:32kTool Calling:SupportedPublished:Aug 31, 2026License:apache-2.0Architecture:Transformer Open Weights Featherless Exclusive Cold

The cosmos1030/gmp-kd3e-1-s70pct-lr1e-4_20260901_080954 model is a 4 billion parameter Qwen3-4B variant, developed by cosmos1030, that has been pruned to 70% unstructured sparsity using the SCOUT method. This model achieves an average score of 45.58 across five tasks including MATH-500 and GSM8K, demonstrating efficient performance for a highly sparse model. It utilizes a projected-gradient mask update strategy with a self-KL trust region and a combined next-token, offline-KD, and on-policy-KD objective for weight recovery. This model is optimized for maintaining performance at significant sparsity levels, making it suitable for research into efficient model deployment.

Loading preview...

Model Overview

This model, cosmos1030/gmp-kd3e-1-s70pct-lr1e-4_20260901_080954, is a 4 billion parameter Qwen3-4B base model that has been pruned to 70% unstructured sparsity using the SCOUT method. Developed by cosmos1030, this specific repository represents the 'headline arm' (A0) of an ablation study, which achieved the best score among its 70% sparse counterparts.

Key Characteristics & Performance

  • Sparsity: Achieves 70% unstructured sparsity, meaning 70% of its weights are zeroed out.
  • Pruning Method: Employs SCOUT, which uses projected-gradient mask updates gated by a measured self-KL trust region, combined with a recovery objective including next-token prediction (NTP), offline Knowledge Distillation (KD), and on-policy KD.
  • Performance: Achieved an avg5 score of 45.58 across a quick-profile lighteval benchmark, including:
    • MATH-500: 72.40 accuracy
    • GSM8K: 71.49 extractive_match
    • GPQA-Diamond: 35.86 gpqa_pass@k:k=1
    • IFEval: 39.93 prompt_level_strict_acc
    • LiveCodeBench (codegen): 8.21 codegen_pass@1:16
  • Context Length: Supports a context length of 32768 tokens.

Limitations

  • No inherent speedup: Due to unstructured sparsity and storage as dense safetensors, wall-clock or memory gains require sparse kernels or structured variants.
  • Single run evaluation: Performance metrics are based on a single run with seed 42, and small differences (e.g., <1 point) may not be statistically significant.
  • Evaluation protocol: Scores are comparable within this ablation family but not necessarily across different evaluation harnesses or settings.
  • Inherited limitations: Carries the capabilities, biases, and failure modes of the dense Qwen/Qwen3-4B base model.

Finetunes
1 models