majentik/gemma-4-E4B-turboquant

VISIONConcurrent Unit Cost:1Model Size:7.9BQuant:FP8Context Size:32kTool Calling:SupportedPublished:Apr 6, 2026License:apache-2.0Architecture:Transformer0.0K Open Weights Featherless Exclusive Cold

The majentik/gemma-4-E4B-turboquant model is a documentation repository for applying TurboQuant KV cache compression to Google's gemma-4-E4B, a 7.9 billion parameter dense transformer. This technique reduces attention cache memory during inference, offering up to 8x memory savings for long sequences without modifying the base model weights. It is designed to enhance inference efficiency, particularly for models with large context lengths up to 128K tokens, making it ideal for memory-constrained environments.

Loading preview...

majentik/gemma-4-E4B-turboquant: KV Cache Compression for Gemma-4-E4B

This repository provides documentation and guidance for integrating TurboQuant KV cache compression with the google/gemma-4-E4B model. Unlike traditional weight quantization, TurboQuant is applied at runtime, allowing the same base model weights to benefit from significant memory reductions during inference, especially crucial for handling long context lengths (up to 128K tokens).

Key Capabilities & Features

  • Runtime KV Cache Compression: Reduces attention memory footprint during inference, offering up to 8x memory savings for long sequences.
  • Weight-Agnostic: Works with any weight variant of the base gemma-4-E4B model, as compression is applied dynamically.
  • Efficiency for Long Context: Particularly beneficial for scenarios requiring large context windows by optimizing KV cache usage.
  • Python Integration: Easily applied via the turboquant Python package with Hugging Face transformers.
  • llama.cpp Fork Support: Compatible with a specialized llama-cpp-turboquant fork for C++ environments.

When to Use This

This approach is ideal for developers looking to:

  • Reduce VRAM/RAM usage during inference, especially on devices with limited memory.
  • Improve inference speed (up to 1.7x on some hardware) by optimizing KV cache access.
  • Utilize the gemma-4-E4B model with its 128K context length more efficiently.
  • Combine with existing weight quantization (e.g., GGUF, MLX) for maximum overall efficiency.

Finetunes
4 models