minjaechoi/nemotron3-nano-30b-a3b-2p03bit-r34
The minjaechoi/nemotron3-nano-30b-a3b-2p03bit-r34 is an internal research checkpoint based on the NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 model. This 30 billion parameter model utilizes a routed-expert architecture, achieving an average of 2.025 bits per weight while other weights are in BF16 format. It is designed for efficient deployment with standard transformers and vLLM libraries, making it suitable for applications requiring a balance of performance and reduced memory footprint.
Loading preview...
Model Overview
This model, minjaechoi/nemotron3-nano-30b-a3b-2p03bit-r34, is an internal research checkpoint derived from the nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 base model. It features a 30 billion parameter architecture with a focus on quantization for efficiency.
Key Characteristics
- Quantization: Employs a routed-expert quantization scheme, achieving an average of 2.025 bits per weight. Non-routed weights are stored in BF16 format.
- Compatibility: Weights are stored dequantized in BF16 tensors, ensuring seamless loading and operation with standard
transformersandvLLMlibraries. - Efficiency: The quantization approach aims to reduce the model's memory footprint and potentially improve inference speed while maintaining performance.
Use Cases
This model is particularly well-suited for developers and researchers interested in:
- Efficient Deployment: Leveraging a large language model with reduced memory requirements for resource-constrained environments.
- Quantization Research: Exploring the practical application and performance of routed-expert quantization techniques.
- NVIDIA Nemotron-3 Nano Exploration: Utilizing a quantized variant of the NVIDIA Nemotron-3 Nano 30B-A3B series for various NLP tasks.