Rishidar/autoscientist-language-qlora
Rishidar/autoscientist-language-qlora is a 0.5 billion parameter causal language model, fine-tuned from Qwen/Qwen2.5-0.5B-Instruct using QLoRA. Developed by Rishidar, this model is specifically adapted for language tasks within the AutoScientist Challenge. It excels at processing and generating language based on a high-quality, specialized dataset.
Loading preview...
AutoScientist Language Model
This model, Rishidar/autoscientist-language-qlora, is a 0.5 billion parameter language model fine-tuned from the Qwen/Qwen2.5-0.5B-Instruct base model. Its primary distinction lies in its specialized training for the Adaption Labs AutoScientist Challenge, utilizing a unique "language adapted dataset" from Rishidar/autoscientist-competition-datasets.
Key Training Details
- Methodology: Fine-tuned using QLoRA (Quantized Low-Rank Adaptation) with 4-bit NF4 quantization.
- Configuration: Employed a rank (r) of 32 and an alpha value of 64.
- Epochs: Trained for 3 epochs.
- Learning Rate: A learning rate of 0.0002 was used during training.
- Dataset Quality: The training dataset received a "Grade A" evaluation from Adaption Labs, indicating high quality and relevance for its intended purpose.
Intended Use
This model is specifically designed for tasks related to the AutoScientist Challenge, where understanding and generating language pertinent to scientific contexts or problem-solving is crucial. Its compact size (0.5B parameters) combined with a substantial context length of 32768 tokens makes it efficient for specialized language processing within its domain.