IAmSkyDra/SEA-Instruct-URA-LLaMA-2.1-8B
IAmSkyDra/SEA-Instruct-URA-LLaMA-2.1-8B is an 8 billion parameter instruction-tuned causal language model, developed by IAmSkyDra. It is a merged LoRA fine-tune of ura-hcmut/ura-llama-2.1-8b using the SEA-Instruct-2602 dataset, optimized for instruction following. The model incorporates the native Llama 3.1 chat template and is suitable for research in instruction-tuned LLMs.
Loading preview...
Model Overview
IAmSkyDra/SEA-Instruct-URA-LLaMA-2.1-8B is an 8 billion parameter instruction-tuned language model. It is derived from ura-hcmut/ura-llama-2.1-8b through a LoRA (Low-Rank Adaptation) supervised fine-tuning process. The training utilized the trannguyenquynhnhu/SEA-Instruct-2602-fine-tuned dataset, aiming to enhance its instruction-following capabilities.
Key Training Details
- Base Model:
ura-hcmut/ura-llama-2.1-8b - Fine-tuning Method: LoRA, with the adapter merged into the base model.
- Framework: Open Instruct
- Precision: BF16
- LoRA Configuration: Rank 16, Alpha 32, Dropout 0.05
- Context Length: Trained with a context length of 4096 tokens.
- Dataset Size: 508,838 training examples after filtering.
- Epochs: 1
- Learning Rate: 5e-6 with a linear scheduler and 3% warmup.
Usage and Considerations
The model includes the native Llama 3.1 chat template for easy integration. It is primarily intended for research purposes. Users should conduct thorough evaluations regarding accuracy, safety, and cultural correctness before considering deployment in any application.