nvidia/Llama-3.1-Minitron-4B-Depth-Base
The NVIDIA Llama-3.1-Minitron-4B-Depth-Base is a 4 billion parameter base text-to-text model, derived from Llama-3.1-8B through pruning of transformer blocks and subsequent continued training with distillation on 94 billion tokens. Developed by NVIDIA, this model utilizes a Llama-3.1 architecture with Grouped-Query Attention and Rotary Position Embeddings, making it suitable for a variety of natural language generation tasks. It is optimized for efficiency through its depth-pruned design while maintaining a 32768 token context length, and is ready for commercial use under the NVIDIA Open Model License. The model's training included English, multilingual text, and code, with a data cutoff of June 2023.
Loading preview...
Model Overview
NVIDIA's Llama-3.1-Minitron-4B-Depth-Base is a 4 billion parameter base text-to-text model, engineered for diverse natural language generation tasks. It was developed by NVIDIA through a unique process of depth-pruning from the larger Llama-3.1-8B model, specifically reducing the number of transformer blocks. Following pruning, the model underwent continued training with distillation using 94 billion tokens, leveraging the continuous pre-training data corpus from Nemotron-4 15B. This approach aims to create a more compact yet capable model, as detailed in their technical report.
Key Capabilities & Architecture
- Architecture: Based on the Llama-3.1 network, featuring a Transformer Decoder with 32 layers, 4096 embedding size, 32 attention heads, and Grouped-Query Attention (GQA) with Rotary Position Embeddings (RoPE).
- Training Data: Trained on a diverse corpus including English, multilingual text, and code, covering domains like legal, math, science, and finance, with a data cutoff of June 2023.
- Context Length: Supports a context length of 32768 tokens, performing well with inputs up to 8k characters.
- Commercial Use: Released under the NVIDIA Open Model License Agreement and is ready for commercial applications.
Performance Highlights
- Language Understanding (MMLU 5-shot): Achieves an average score of 58.7.
- Reasoning & Common Sense (Zero-shot): Scores include 73.2 on HellaSwag, 72.1 on Winogrande, and 52.6 on ARC-Challenge.
- Code Generation (MBPP): Demonstrates a score of 30.7.
Limitations
Like many models trained on internet data, Llama-3.1-Minitron-4B-Depth-Base may exhibit biases, generate toxic or inaccurate responses, or include irrelevant information. Users should be aware of these potential issues and implement appropriate safeguards.