minjaechoi/nemotron3-nano-30b-a3b-2p25bit-r17
The minjaechoi/nemotron3-nano-30b-a3b-2p25bit-r17 model is an internal research checkpoint based on NVIDIA's Nemotron-3-Nano-30B-A3B-BF16 architecture, featuring 30 billion parameters. This model utilizes a unique routed-expert quantization scheme, achieving an average of 2.254 bits per weight while maintaining BF16 precision for other weights. It is designed for efficient inference by storing weights dequantized in BF16 tensors, compatible with standard transformers and vLLM libraries.
Loading preview...
Model Overview
The minjaechoi/nemotron3-nano-30b-a3b-2p25bit-r17 is an internal research checkpoint derived from the NVIDIA Nemotron-3-Nano-30B-A3B-BF16 base model. This 30-billion parameter language model incorporates a specialized quantization technique, where routed experts achieve an average of 2.254 bits per weight, while all other weights are maintained in BF16 format. The model's design allows for efficient loading and inference, as weights are stored dequantized in BF16 tensors, ensuring compatibility with popular libraries like transformers and vLLM.
Key Characteristics
- Base Model: Built upon
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16. - Parameter Count: 30 billion parameters.
- Quantization: Features a routed-expert quantization scheme with an average of 2.254 bits per weight.
- Mixed Precision: Combines 2.254-bit routed experts with BF16 precision for other weights.
- Compatibility: Designed to load seamlessly with standard
transformersandvLLMlibraries, as weights are stored dequantized in BF16.
Potential Use Cases
This model is particularly suited for scenarios where:
- Efficient Inference: The mixed-precision quantization aims to reduce memory footprint and accelerate inference speed compared to full BF16 models, without significant performance degradation.
- Research and Development: As an internal research checkpoint, it offers a platform for exploring advanced quantization techniques and their impact on large language models.
- Hardware Optimization: Its design could be beneficial for deployment on hardware platforms that can leverage mixed-precision operations effectively.