minjaechoi/nemotron3-nano-30b-a3b-gsq-2p25bit-r35
The minjaechoi/nemotron3-nano-30b-a3b-gsq-2p25bit-r35 is a 30 billion parameter internal research checkpoint from NVIDIA's Nemotron-3-Nano family. This model utilizes a unique quantization scheme with routed experts averaging 2.25 bits, where every other weight is in BF16 format. It employs a GSQ baseline with group size 64 and 2-bit packed codes with BF16 group scales, designed for efficient inference while maintaining performance. The model's weights are stored dequantized in BF16 tensors, ensuring compatibility with standard `transformers` and vLLM libraries.
Loading preview...
Model Overview
The minjaechoi/nemotron3-nano-30b-a3b-gsq-2p25bit-r35 is an internal research checkpoint from the NVIDIA Nemotron-3-Nano 30 billion parameter model family. This particular iteration, identified as r35, focuses on advanced quantization techniques to optimize model size and inference efficiency.
Key Characteristics
- Parameter Count: 30 billion parameters.
- Quantization Scheme: Features routed experts with an average of 2.25 bits per weight. This is achieved by quantizing some weights to 2 bits while others remain in BF16 format.
- GSQ Baseline: Implements a Group-Size Quantization (GSQ) baseline with a group size of 64, utilizing 2-bit packed codes and BF16 group scales.
- Weight Storage: Weights are stored dequantized in BF16 tensors, allowing for seamless loading and usage with standard libraries like
transformersand vLLM. - Efficiency: The mixed-precision approach (2.25 bits average) aims to provide a balance between model performance and computational resource requirements.
Intended Use Cases
This model is primarily a research checkpoint demonstrating advanced quantization strategies. It is suitable for:
- Research and Development: Exploring the impact of highly efficient quantization methods on large language models.
- Efficient Deployment: Potentially serving as a foundation for applications requiring reduced memory footprint and faster inference speeds compared to full-precision models, while maintaining reasonable performance.
- Compatibility Testing: Verifying the integration of models with complex quantization schemes into existing inference frameworks.