Naturemort/Gemma-3-270m-it-GroomAttention

Naturemort/Gemma-3-270m-it-GroomAttention: Experimental Attention Modification

This model is an experimental MLX checkpoint derived from google/gemma-3-270m-it, maintaining the original Gemma 3 270M architecture and dense tensor shapes. Its core innovation lies in replacing the first transformer layer's self-attention projection weights with a "GroomAttention" block-modified variant, identified through evolutionary search.

Key Characteristics & Modifications

• Targeted Modification: Only the query, key, value, and output projections of the first self-attention layer were altered. All other layers, tokenizer behavior, and architectural settings remain consistent with the base Gemma model.
• Dense-Compatible Weight Modification: GroomAttention is not a new attention kernel or a structural pruning. It's a modification of learned weights where original dense projection matrices are treated as grids of blocks, with each block undergoing a selected numeric operation (e.g., suppression, rescaling, rounding to lower precision). The resulting matrices retain their original dense shape.
• Behavioral Shift: This modification changes the learned geometry within the first attention block, influencing which token-to-token relations receive high attention scores and how information is carried and remixed. Diagnostic prompts show non-uniform entropy changes across attention heads and shifts in token-to-token relation strengths.
• Compression Gain (Internal): An internal search reported an estimated block-representation compression gain of about 24.74% for the searched attention-module representation, though the released checkpoint remains a standard dense MLX-compatible model.

Performance & Limitations

• Benchmark Impact: On a custom 0-shot multiple-choice continuation evaluation, the model largely preserves the base model's behavior, with an average accuracy drop of 0.0069 across HellaSwag, PIQA, ARC Challenge, and WinoGrande. WinoGrande showed a slight improvement.
• Experimental Nature: This is an experimental checkpoint for research into attention mechanisms. The evaluation results are from a custom local scorer and are not directly comparable to official Google Gemma benchmarks.
• No Physical Size Reduction: Despite internal compression gains, the released model is dense and does not offer a smaller physical model size.