exolabs/Qwen3.5-9B-Q4KM-dequant-bf16-vllm
The exolabs/Qwen3.5-9B-Q4KM-dequant-bf16-vllm model is a 9 billion parameter Qwen3.5 variant, specifically a dequantized BF16 version of the Q4_K_M GGUF checkpoint, optimized for deployment with vLLM. This model is engineered for efficient inference, leveraging BF16 precision and vLLM's Triton attention and GDN prefill backends. It is suitable for general language model tasks, demonstrated by basic smoke tests for factual recall and arithmetic.
Loading preview...
exolabs/Qwen3.5-9B-Q4KM-dequant-bf16-vllm: Optimized for vLLM Inference
This model is a specialized 9 billion parameter variant of the Qwen3.5 series, meticulously prepared by Exolabs for high-performance inference using vLLM. It originates from a unsloth/Qwen3.5-9B-GGUF Q4_K_M checkpoint, which has been dequantized to FP32 and then cast to BF16 precision.
Key Optimizations & Features
- Dequantized BF16 Precision: The model's weights were dequantized from Q4_K_M GGUF to FP32 and then converted to BF16, balancing performance and memory efficiency for vLLM.
- vLLM Integration: Specifically configured and validated for vLLM 0.23.0, utilizing
bfloat16dtype,triton_attnattention backend, andtritonGDN prefill backend for optimized throughput. - Architecture Alignment: Configuration, tokenizer, chat template, and generation settings are derived from the upstream
Qwen/Qwen3.5-9Brepository, ensuring compatibility with the original Qwen3.5 architecture. - Efficient Deployment: Designed for serving with vLLM, enabling efficient handling of requests and leveraging hardware acceleration.
Use Cases
This model is well-suited for general language model applications where efficient inference and BF16 precision are beneficial. Basic smoke tests confirm its ability to handle common tasks like factual recall and simple arithmetic, making it a robust choice for various text generation and understanding applications within a vLLM serving environment.