AstroMLab/astrollama-2-70b-base_aic
AstroMLab/astrollama-2-70b-base_aic is a 69 billion parameter specialized base language model developed by AstroMLab, fine-tuned from Meta's LLaMA-2-70b architecture. It is uniquely trained on Abstract, Introduction, and Conclusion sections of arXiv's astro-ph papers, making it the first specialized 70B-level LLM for astronomy. This model excels at next token prediction for astronomy-related text generation and analysis, demonstrating improved performance over its LLaMA-2-70B baseline on astronomical Q&A benchmarks.
Loading preview...
AstroLLaMA-2-70B-Base_AIC: A Specialized Astronomical LLM
AstroLLaMA-2-70B-Base_AIC is a 69 billion parameter base language model developed by the AstroMLab team, representing a significant advancement in specialized LLMs for astronomy. It is built upon Meta's LLaMA-2-70b architecture and has undergone Continual Pre-Training (CPT) using the LMFlow framework.
Key Features and Training:
- Specialized Training Data: Fine-tuned exclusively on Abstract, Introduction, and Conclusion (AIC) sections from arXiv's astro-ph category papers, covering all publications up to July 2023. This unique dataset focuses the model on the core scientific discourse within astronomy.
- Architecture: Based on the robust LLaMA-2-70b model, providing a strong foundation for specialized knowledge.
- Training Details: Fine-tuned for approximately 2,000 A100 GPU hours with a learning rate of 2 × 10⁻⁵ and a maximum token length of 2048.
- Primary Function: Designed for next token prediction tasks, making it suitable for generating and analyzing astronomy-related text. It is not an instruct or chat model.
Performance and Significance:
This model demonstrates notable improvements over its general-purpose LLaMA-2-70B baseline. On the astronomical benchmarking Q&A described in Ting et al. 2024, AstroLLaMA-2-70B-Base_AIC achieved a score of 76.0%, outperforming LLaMA-2-70B (70.7%), LLaMA-3.1-8B (73.7%), and other general LLMs in its class. This highlights the effectiveness of specialized training for large language models.
Ethical Considerations:
Users should be aware of the potential for generating misleading scientific content and are advised to verify model outputs against peer-reviewed sources for critical applications.
When to Use This Model:
- Astronomy Text Generation: Ideal for generating coherent and contextually relevant text within the domain of astrophysics.
- Scientific Text Analysis: Useful for tasks requiring an understanding of astronomical literature, such as summarizing or extracting information from scientific papers.
- Research and Development: A valuable tool for researchers and developers working on applications that require deep domain-specific knowledge in astronomy.