jiamingshan/AHA-L2A-Qwen3-1.7B-repro

TEXT GENERATIONConcurrent Unit Cost:1Model Size:2BQuant:BF16Context Size:32kTool Calling:SupportedPublished:Jul 14, 2026License:apache-2.0Architecture:Transformer Open Weights Featherless Exclusive Cold

The jiamingshan/AHA-L2A-Qwen3-1.7B-repro model is a 1.7 billion parameter Qwen3-based language model designed for reproducing research on sparse attention mechanisms. This package specifically compares two sparse-attention variants, AHA and an L2A-style shared-gate, both derived from the same tuned Qwen3-1.7B checkpoint. It provides a self-contained environment for training and evaluating these variants, focusing on the impact of router granularity on model performance and sparsity. The model is primarily for research and reproducibility in sparse attention and efficient LLM architectures.

Loading preview...

Overview

This repository provides a self-contained package for reproducing research on sparse attention mechanisms, specifically comparing two variants: AHA and an L2A-style shared-gate, both based on the Qwen3-1.7B architecture. The key experimental variable is the router granularity, with AHA using one gate per token, KV head, and layer, while the L2A-style variant uses one gate per token and layer, shared by every head. The package includes the tuned-vanilla starting checkpoint and a two-stage training protocol.

Key Features & Protocol

  • Reproducibility: Designed for exact reproduction of the experimental results, including pinned Docker environment, verified file hashes, and dataset checks.
  • Sparse Attention Variants: Compares AHA (token_kv_head granularity) and an L2A-style shared-gate (token granularity) using the same underlying AHA code, local attention, losses, and optimizer settings.
  • Training Protocol: Employs a two-stage training process: Stage 1 for gate distillation and Stage 2 for grouped-LR recovery, starting from a tuned-vanilla checkpoint with dynamic gate weights initialized to full attention.
  • Evaluation: Strict evaluation on a suite of tasks including local RULER, BabiLong, HELMET-ICL, and MRCR, scanning various inference thresholds to identify the highest effective sparsity maintaining at least 95% of tuned vanilla's macro score.
  • Hardware Requirements: Requires at least one NVIDIA GPU with 80 GB of memory (e.g., A100 80 GB) and 120 GB of free disk space.

Intended Use

This package is ideal for researchers and developers interested in:

  • Reproducing the specific sparse attention experiments detailed in the repository.
  • Understanding the impact of different router granularities (AHA vs. L2A-style) on Qwen3-1.7B performance.
  • Studying the training and evaluation methodologies for sparse LLMs.
  • Exploring efficient LLM architectures and their practical implementation.

Finetunes
1 models