Quatfit/Quatfit-Mini-FP8

Quatfit Mini — FP8: High-Throughput Multimodal Inference

This repository features an FP8-quantized build of Quatfit Mini, an 8 billion parameter multimodal model based on the Google Gemma 4 architecture, supporting a 131,072 token context length. It is specifically designed for high-throughput, GPU-served inference on FP8-capable NVIDIA hardware (Hopper, Ada Lovelace, Blackwell generations).

Key Characteristics & Optimizations

• FP8 Quantization: Utilizes FP8 (OCP E4M3) for the text decoder's linear layers, significantly reducing memory footprint (~9.5 GB VRAM vs. ~16 GB for BF16) and increasing compute throughput by approximately 1.6–1.9x on supported GPUs.
• Multimodal: Supports text, image, and audio modalities, with vision and audio encoders kept at BF16 precision for higher fidelity.
• Quantization Recipe: Employs per-channel static weight scales and per-tensor dynamic activation scales, calibrated with a representative dataset, ensuring minimal quality degradation (FP8 scores match FP32 within 0.3% noise).
• Prompt Format: Uses a Gemma 4-style turn format, including support for tool calling and an optional thinking mode for extended reasoning.

Ideal Use Cases

• Server-side Deployment: Best suited for runtimes like vLLM, TensorRT-LLM, and SGLang, where maximizing inference throughput and minimizing VRAM usage on modern NVIDIA GPUs are critical.
• High-Volume Inference: Provides substantial performance gains for applications requiring rapid processing of multimodal inputs.
• Specific Hardware: Optimized for NVIDIA Hopper, Ada Lovelace, and Blackwell architectures to leverage native FP8 tensor-core acceleration.