Model Overview
CHIH-HUNG/llama-2-13b-OpenOrca_5w is a 13 billion parameter language model developed by CHIH-HUNG. It is a fine-tuned version of the meta-llama/Llama-2-13b-hf base model, specifically trained on the initial 50,000 entries of the OpenOrca dataset.
Fine-Tuning Details
The model was fine-tuned using LoRA (Low-Rank Adaptation) with a rank of 8, targeting the q_proj and v_proj layers. The training utilized a single RTX4090 GPU, a batch size of 8, and a learning rate of 5e-5 for one epoch, employing bf16 precision and 4-bit quantization. The training process recorded a loss of 0.903261117822906 over approximately 7 hours and 20 minutes.
Performance Benchmarks
Evaluations against the HuggingFaceH4/open_llm_leaderboard show improved performance compared to the base Llama-2-13b model across several benchmarks:
- Average Score: 61.2 (compared to 56.9 for Llama-2-13b-hf)
- ARC: 61.01
- HellaSwag: 82.82
- MMLU: 56.09
- TruthfulQA: 44.87
These results indicate that the fine-tuning on the OpenOrca dataset has enhanced the model's reasoning, common sense, and factual recall abilities.
Intended Use
This model is suitable for applications requiring improved performance in general knowledge, reasoning, and question-answering tasks, leveraging the strengths gained from the OpenOrca instruction-tuning dataset.