minjaechoi/qwen3p6-35b-a3b-2p03bit-r75
minjaechoi/qwen3p6-35b-a3b-2p03bit-r75 is a 35.1 billion parameter language model based on the Qwen3.6-35B-A3B architecture. This model features a unique quantization scheme where routed experts average 2.0331 bits, while other weights remain in BF16 format. It is designed for efficient loading with standard `transformers` and vLLM libraries, making it suitable for users seeking a balance between performance and reduced memory footprint through advanced quantization.
Loading preview...
Overview
minjaechoi/qwen3p6-35b-a3b-2p03bit-r75 is an internal research checkpoint derived from the Qwen/Qwen3.6-35B-A3B base model. This 35.1 billion parameter model incorporates a specialized quantization technique to optimize its footprint and efficiency. It is designed to be compatible with standard inference libraries like transformers and vLLM, allowing for straightforward integration into existing workflows.
Key Characteristics
- Base Model: Built upon the robust Qwen/Qwen3.6-35B-A3B architecture.
- Advanced Quantization: Utilizes a unique routing mechanism where expert weights are quantized to an average of 2.0331 bits. All other weights are maintained in BF16 precision.
- Memory Efficiency: This mixed-precision approach aims to reduce the model's memory footprint while preserving performance, as weights are stored dequantized in BF16 tensors.
- Standard Compatibility: Loads seamlessly with stock
transformersand vLLM, ensuring ease of use for developers.
Good for
- Researchers and developers exploring advanced quantization techniques and their impact on large language models.
- Applications where memory optimization is critical, but a high-performing 35B parameter model is desired.
- Experimenting with efficient deployment of large models without complex custom loading procedures.