MiniLLM/Pretrain-Qwen-500M
TEXT GENERATIONPricing:Input $0.04 / Cached $0.002 / Output $0.08Concurrent Unit Cost:1Model Size:0.6BQuant:BF16Context Size:32kTool Calling:SupportedPublished:Oct 19, 2024License:apache-2.0Architecture:Transformer Open Weights Featherless Exclusive Warm
MiniLLM/Pretrain-Qwen-500M is a 0.6 billion parameter language model based on the Qwen architecture, conventionally pre-trained from scratch by MiniLLM. It was trained on The Pile dataset for 50 billion tokens, serving as a foundational baseline model. This model is specifically designed as a pre-training component for the MiniLLM-Qwen-500M series, demonstrating strong performance scaling across model sizes.
Loading preview...
Overview
MiniLLM/Pretrain-Qwen-500M is a 0.6 billion parameter model utilizing the Qwen architecture. Developed by MiniLLM, it was pre-trained from scratch on The Pile dataset, consuming 50 billion tokens during its training phase. This model serves as a crucial baseline for the subsequent MiniLLM-Qwen-500M series.
Key Characteristics
- Architecture: Qwen-based, designed for efficient language processing.
- Training Data: Pre-trained on the comprehensive Pile dataset.
- Reproducibility: MiniLLM provides the tokenized pre-training corpus to ensure research reproducibility.
- Performance: MiniPLM models, including this baseline, are noted for achieving improved performance relative to computational resources and exhibit strong scaling across different model sizes.
Use Cases
- Foundation Model: Ideal as a starting point for further fine-tuning or distillation tasks.
- Research Baseline: Suitable for researchers exploring knowledge distillation or pre-training methodologies, as detailed in the associated paper.
- Comparative Analysis: Can be used to compare against other baselines like VanillaKD-Pretrain-Qwen-500M.