minjaechoi/nemotron3-nano-30b-a3b-2p02bit-r19
minjaechoi/nemotron3-nano-30b-a3b-2p02bit-r19 is a 30 billion parameter language model based on the NVIDIA Nemotron-3 Nano architecture. This model is an internal research checkpoint featuring a unique quantization scheme where routed experts average 2.0217 bits, while other weights remain in BF16 format. It is designed for efficient deployment, loading dequantized weights directly with standard transformers and vLLM libraries.
Loading preview...
Model Overview
minjaechoi/nemotron3-nano-30b-a3b-2p02bit-r19 is an internal research checkpoint derived from the NVIDIA Nemotron-3 Nano-30B-A3B-BF16 base model. This 30 billion parameter language model incorporates a specialized quantization technique to optimize its footprint and performance.
Key Characteristics
- Quantization Scheme: Utilizes a routed experts approach where the average bit-width for these experts is 2.0217 bits. All other weights are maintained in BF16 (BFloat16) format.
- Weight Handling: Despite the quantization, weights are stored dequantized in BF16 tensors, allowing for seamless loading and operation with standard libraries such as
transformersandvLLMwithout requiring custom loading mechanisms. - Base Model: Built upon the
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16architecture. - Internal ID: Identified as
r19, indicating its specific research iteration.
Use Cases
This model is particularly suited for research and development environments focused on:
- Efficient Deployment: Leveraging its unique quantization for reduced memory footprint and potentially faster inference.
- Experimentation with Quantization: Exploring the impact and performance of mixed-precision quantization strategies.
- NVIDIA Nemotron-3 Ecosystem: Integrating with existing workflows and tools designed for the Nemotron-3 family of models.