ClimateGPT-7B is a 7 billion parameter transformer decoder model developed by eci-io, adapted from Llama-2 and continuously pre-trained on 4.2 billion tokens of curated climate documents. This model is specifically fine-tuned for climate science research and question answering, outperforming Llama-2-70B Chat on climate-specific benchmarks. With a 4K token context length, it is optimized for synthesizing interdisciplinary climate research and providing specialized feedback for decision-makers, scientists, and journalists.
Loading preview...
ClimateGPT-7B Overview
ClimateGPT-7B is a 7 billion parameter, decoder-only Transformer model developed by eci-io, in collaboration with Erasmus AI and AppTek. It is an adaptation of Llama-2-7B, continuously pre-trained on a substantial dataset of 4.2 billion tokens derived from curated climate documents. The model has been further instruction fine-tuned on approximately 272,000 instruction-completion pairs, focusing on both climate-specific and general domains.
Key Capabilities & Differentiators
- Climate Science Specialization: Designed to synthesize interdisciplinary research on climate change, making it highly specialized for the climate domain.
- Superior Climate Performance: Outperforms Llama-2-70B Chat on climate-specific benchmarks, indicating strong domain expertise despite its smaller size.
- Instruction-Tuned for QA: Optimized for direct use in climate-specific question-answering applications.
- Retrieval Augmentation Support: Built to work effectively with retrieval augmentation, supporting up to 5 references in context to enhance knowledge and factuality.
- Context Length: Features a 4K token context length, suitable for processing detailed climate-related queries.
Ideal Use Cases
- Specialized Question Answering: Directly applicable for answering questions within the climate domain.
- Decision Support: Provides useful feedback for decision-makers, scientists, and journalists involved in climate discussions.
- Further Fine-tuning: Serves as a strong base model for developers interested in further fine-tuning for specific climate-related tasks.
It is important to note that ClimateGPT-7B is not intended as a general-purpose chatbot, though it possesses chat capabilities. The model is designed to be integrated with retrieval augmentation and cascaded machine translation for extended knowledge and language coverage, as demonstrated on the eci.io platform.