JingyaoLi/ScienceLLaMA-1b
JingyaoLi/ScienceLLaMA-1b is a 1 billion parameter language model fine-tuned by JingyaoLi using Logits-Based Finetuning on the Science-Logits-1.2M dataset. This method combines supervised learning with knowledge distillation, integrating teacher logits and ground truth labels to enhance both correctness and linguistic diversity. The model is designed for scientific domain applications, leveraging its specialized training to improve performance in relevant tasks. It features a context length of 32768 tokens.
Loading preview...
ScienceLLaMA-1b Overview
ScienceLLaMA-1b is a 1 billion parameter language model developed by JingyaoLi, specifically fine-tuned for scientific applications. It utilizes a novel Logits-Based Finetuning approach, which merges the benefits of supervised learning with knowledge distillation. This technique incorporates teacher logits alongside ground truth labels from the JingyaoLi/Science-Logits-1.2M dataset, aiming to preserve both factual accuracy and diverse linguistic expression.
Key Capabilities
- Enhanced Scientific Understanding: Specialized training on a science-focused dataset improves its ability to process and generate scientific text.
- Logits-Based Finetuning: Leverages a unique training methodology that combines explicit supervision with knowledge transfer from a teacher model.
- Linguistic Diversity: Designed to maintain varied language generation while ensuring factual correctness.
Good for
- Applications requiring language understanding and generation within scientific domains.
- Researchers and developers looking for a compact model with specialized scientific knowledge.
- Tasks benefiting from a model trained with a blend of supervised learning and knowledge distillation techniques.