openpangu/openPangu-7B-Diffusion-Base
openPangu-7B-Diffusion-Base is a 7 billion parameter dense language model developed by openPangu, utilizing a novel context-causal block diffusion mechanism. This model is pretrained with an 8k context length and excels in general, mathematical, and coding benchmarks, demonstrating strong performance across MMLU, CEVAL, GSM8K, and MATH. Its unique diffusion-based architecture allows for flexible context lengths and improved inference throughput, making it suitable for applications requiring efficient and high-performance language generation.
Loading preview...
Overview
openPangu-7B-Diffusion-Base is a 7 billion parameter dense language model developed by openPangu, distinguished by its innovative context-causal block diffusion mechanism. This architecture enables efficient block-wise decoding with flexible context lengths, offering significant throughput improvements compared to standard autoregressive models. The model was pretrained on 700B tokens with an 8k context length, followed by annealing to 32k context length and 10 epochs of 10B slow-thought SFT for the openPangu-R-7B-Diffusion variant.
Key Capabilities
- Context-Causal Block Diffusion: Employs a novel decoding strategy that processes tokens in blocks, combining full attention within blocks and causal attention for preceding context. This allows for variable-length inference and efficient KV caching.
- Flexible Context Length: Supports flexible context lengths, not limited by block size, and offers both autoregressive and block diffusion decoding modes.
- Enhanced Throughput: Achieves up to 2.5x higher throughput compared to standard autoregressive decoding through confidence threshold sampling.
- Optimized Training: The training methodology efficiently utilizes both masked and unmasked context parts, ensuring stable token participation in batches, especially for long sequences.
Good For
- High-Performance Inference: Ideal for applications requiring fast and efficient language generation due to its block diffusion decoding and optimized sampling.
- General Language Tasks: Demonstrates strong performance across a range of general benchmarks, including MMLU (70.09%), CEVAL (73.03%), and CMMLU (77.27%), outperforming several comparable models.
- Mathematical and Reasoning Tasks: Excels in mathematical reasoning, scoring 78.77% on GSM8K and 46.02% on MATH.
- Code Generation: Shows competitive performance in coding benchmarks like MBPP (55.80% Pass@1) and HumanEval (50.00% Pass@1).