wcamon/circus-0.4-t9
wcamon/circus-0.4-t9 is a 4.5 billion parameter language model developed by Wei-Ciao Wu, derived from Agents-A1-4B. It features a novel additive two-plane 9-level quantization grid, constraining every linear weight to achieve approximately 3.67 bits per weight. This model focuses on demonstrating an advanced quantization recovery methodology and the additive bit-plane form for efficient inference, maintaining high retention across various benchmarks like MMLU and GSM8k.
Loading preview...
Overview
wcamon/circus-0.4-t9 is a 4.5 billion parameter language model based on the Agents-A1-4B architecture, developed by Wei-Ciao Wu. Its core innovation lies in its additive two-plane 9-level quantization grid, where every linear weight is constrained to a specific set of values, achieving an approximate density of 3.67 bits per weight. This approach enables the potential for addition-only matmul kernels, similar to BitNet-style models, offering significant inference efficiency.
Key Characteristics
- Advanced Quantization: Utilizes a unique 9-level additive bit-plane quantization scheme, represented by two ternary planes. This allows for a compact representation (1.7 GB true quantized artifact vs. 8.5 GB bf16 materialization).
- High Retention: Despite aggressive quantization, the model demonstrates strong performance retention, achieving over 90% retention on tasks like GSM8k (96.3%), MMLU (99.0%), and HumanEval (93.6%) compared to its bf16 base.
- Novel Recovery Methodology: The model was produced using a sophisticated recovery recipe involving GPTQ-style per-layer initialization, sliding-window soft-anneal reconstruction, and light polish with scale-only logit KD. This methodology is a key contribution, particularly for low-bit recovery.
- Dual Artifacts: Ships with both a bf16 materialized version for immediate use with
transformersand abitplanes_k2_c0.6.npzfile representing the true bit-exact quantized artifact.
Good For
- Research in Quantization: Ideal for researchers exploring advanced quantization techniques, particularly additive bit-plane forms and their recovery methodologies.
- Efficient Inference Development: Provides a foundation for developing and testing addition-only matmul kernels for highly efficient, low-bit inference.
- Understanding Low-Bit Model Behavior: Offers insights into the trade-offs and challenges of achieving high performance with very low bit-width models, especially regarding assignment quality and recovery bottlenecks.