minjaechoi/nemotron3-nano-30b-a3b-2p03bit-r21
The minjaechoi/nemotron3-nano-30b-a3b-2p03bit-r21 is a 30 billion parameter language model based on the NVIDIA Nemotron-3 Nano architecture, featuring an internal research checkpoint with routed experts. This model utilizes an average of 2.0251 bits for its routed experts, while other weights are stored in BF16 format. It is designed for efficient loading with standard transformers and vLLM libraries, offering a compact yet capable solution for various language tasks.
Loading preview...
Model Overview
The minjaechoi/nemotron3-nano-30b-a3b-2p03bit-r21 is an internal research checkpoint derived from the NVIDIA Nemotron-3 Nano-30B-A3B-BF16 base model. This 30 billion parameter language model incorporates a unique quantization strategy, utilizing "routed experts" that average 2.0251 bits, while all other weights are maintained in BF16 precision.
Key Characteristics
- Quantization: Employs a routed-expert quantization scheme, achieving an average of 2.0251 bits for these specific components.
- Precision: Non-expert weights are stored and loaded in BF16 (BFloat16) format, ensuring a balance between performance and memory footprint.
- Compatibility: Designed to be compatible with standard machine learning libraries such as
transformersandvLLM, allowing for straightforward integration and deployment. - Base Model: Built upon the
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16architecture, inheriting its foundational capabilities.
Use Cases
This model is particularly suited for applications where:
- Resource Efficiency: There is a need for a powerful 30B parameter model with reduced memory requirements due to its mixed-precision quantization.
- Research & Development: Users are interested in exploring models with advanced quantization techniques like routed experts.
- Deployment: Compatibility with common inference frameworks like
transformersandvLLMsimplifies deployment for various language-based tasks.