CHIH-HUNG/llama-2-13b-FINETUNE2_3w
CHIH-HUNG/llama-2-13b-FINETUNE2_3w is a 13 billion parameter Llama-2-based language model fine-tuned by CHIH-HUNG on the huangyt/FINETUNE2 dataset, comprising approximately 30,000 data points. This model was trained using LoRA with a rank of 8, targeting q_proj and v_proj, and demonstrates improved performance on benchmarks like HellaSwag and TruthfulQA compared to the base Llama-2-13b model. It is suitable for tasks requiring enhanced reasoning and factual recall as indicated by its benchmark scores.
Loading preview...
Model Overview
CHIH-HUNG/llama-2-13b-FINETUNE2_3w is a 13 billion parameter language model built upon the meta-llama/Llama-2-13b-hf architecture. It has been fine-tuned by CHIH-HUNG using the huangyt/FINETUNE2 dataset, which consists of approximately 30,000 training examples.
Fine-Tuning Details
- Base Model:
meta-llama/Llama-2-13b-hf - Dataset:
huangyt/FINETUNE2(approx. 30,000 entries) - Method: LoRA (Low-Rank Adaptation) with a rank of 8
- Target Modules:
q_proj,v_proj - Training: Single epoch,
bf16precision,load_in_4bitquantization, utilizing DeepSpeed for a runtime of 3 hours and 27 minutes.
Performance Benchmarks
Evaluated against the HuggingFaceH4/open_llm_leaderboard, this model shows improvements over the base Llama-2-13b model in several key areas:
- Average Score: 58.34 (compared to 56.9 for base Llama-2-13b)
- HellaSwag: 82.32 (compared to 80.97 for base Llama-2-13b)
- TruthfulQA: 38.17 (compared to 34.17 for base Llama-2-13b)
Use Cases
This model is particularly well-suited for applications where improved performance on reasoning and factual question-answering tasks, as reflected in its HellaSwag and TruthfulQA scores, is beneficial. Its fine-tuning on a specific dataset suggests potential for domain-specific applications aligned with the huangyt/FINETUNE2 dataset's content.