ManniX-ITA/gemma-4-31b-he1-it

VISIONPricing:Input $0.48 / Cached $0.1 / Output $1.44Concurrent Unit Cost:2Model Size:31BQuant:FP8Context Size:32kTool Calling:SupportedPublished:May 11, 2026License:gemmaArchitecture:Transformer0.0K Featherless Exclusive Cold

ManniX-ITA/gemma-4-31b-he1-it is a 31 billion parameter Gemma 4 model from ManniX-ITA, featuring a partial head-prune of the first four sliding-attention layers (L0-L3) with 12.5% of Q-heads removed and an lstsq heal of O-projections. This modification, reproducible via OmniMergeKit, enhances multi-domain reasoning and HumanEval+ performance, making it a strong variant for complex reasoning and code generation tasks. It matches or surpasses the base model on key benchmarks like GPQA Diamond and HumanEval+ while maintaining performance on other code and instruction-following evaluations.

Loading preview...

Overview

ManniX-ITA/gemma-4-31b-he1-it is a 31 billion parameter Gemma 4 model, derived from google/gemma-4-31B-it. This version incorporates a unique partial head-pruning technique, specifically targeting the first four sliding-attention layers (L0-L3) by removing 12.5% of Q-heads and applying an lstsq heal to the O-projections. Layers L4-L59 remain byte-identical to the base model. This modification is fully reproducible using OmniMergeKit.

Key Capabilities & Performance

  • Enhanced Reasoning: Shows a notable +3.03 pp gain on GPQA Diamond, indicating improved multi-domain reasoning capabilities compared to the base model.
  • Superior Code Generation: Achieves 93.90% on HumanEval+, a +1.83 pp increase over the base, making it the strongest Gemma 4 variant in its cohort for this challenging code benchmark.
  • Robust Instruction Following: Matches the base model's performance on benchmarks like LCB-medium-55, AIME 2024, IFEval, and HumanEval, demonstrating no regression in these areas.
  • Efficient Pruning: The pruning method uses a mask-mode approach, avoiding reshaping and allowing the model to be loaded by stock transformers without config patches, while achieving effective parameter savings in attention.

When to Use This Model

  • Complex Reasoning Tasks: Ideal for applications requiring strong multi-domain reasoning, as evidenced by its GPQA Diamond performance.
  • Advanced Code Generation: Highly suitable for code generation and understanding, particularly for harder coding challenges like HumanEval+.
  • Resource-Conscious Deployment: While primarily a research artifact for partial pruning, its performance gains without significant regression make it a compelling choice for tasks where these specific capabilities are critical.

Finetunes
2 models