minjaechoi/nemotron3-nano-30b-a3b-2p02bit-r18
The minjaechoi/nemotron3-nano-30b-a3b-2p02bit-r18 is a 30 billion parameter language model based on NVIDIA's Nemotron-3-Nano-30B-A3B-BF16 architecture, featuring a context length of 32768 tokens. This model utilizes a unique quantization scheme where routed experts average 2.0217 bits, while other weights remain in BF16 format. It is an internal research checkpoint designed for efficient inference by storing weights dequantized in BF16 tensors, compatible with standard transformers and vLLM libraries.
Loading preview...
Model Overview
The minjaechoi/nemotron3-nano-30b-a3b-2p02bit-r18 is a 30 billion parameter language model derived from the NVIDIA Nemotron-3-Nano-30B-A3B-BF16 base model. This particular version represents an internal research checkpoint, identified as 'r18', focusing on advanced quantization techniques.
Key Characteristics
- Parameter Count: 30 billion parameters.
- Base Model: Built upon
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16. - Quantization: Employs a novel quantization strategy where 'routed experts' are compressed to an average of 2.0217 bits. All other weights are maintained in BF16 (BFloat16) precision.
- Compatibility: Weights are stored dequantized in BF16 tensors, ensuring seamless loading and operation with standard
transformersandvLLMlibraries. - Context Length: Supports a substantial context window of 32768 tokens.
Unique Differentiator
This model stands out due to its hybrid quantization approach, combining ultra-low-bit routed experts with BF16 precision for other weights. This design aims to optimize model size and inference efficiency while maintaining performance, making it suitable for scenarios where computational resources are a consideration. The license for this model follows that of its NVIDIA base model.