Model Overview

CHIH-HUNG/llama-2-13b-FINETUNE2_3w is a 13 billion parameter language model built upon the meta-llama/Llama-2-13b-hf architecture. It has been fine-tuned by CHIH-HUNG using the huangyt/FINETUNE2 dataset, which consists of approximately 30,000 training examples.

Fine-Tuning Details

• Base Model: meta-llama/Llama-2-13b-hf
• Dataset: huangyt/FINETUNE2 (approx. 30,000 entries)
• Method: LoRA (Low-Rank Adaptation) with a rank of 8
• Target Modules: q_proj, v_proj
• Training: Single epoch, bf16 precision, load_in_4bit quantization, utilizing DeepSpeed for a runtime of 3 hours and 27 minutes.

Performance Benchmarks

Evaluated against the HuggingFaceH4/open_llm_leaderboard, this model shows improvements over the base Llama-2-13b model in several key areas:

• Average Score: 58.34 (compared to 56.9 for base Llama-2-13b)
• HellaSwag: 82.32 (compared to 80.97 for base Llama-2-13b)
• TruthfulQA: 38.17 (compared to 34.17 for base Llama-2-13b)

Use Cases

This model is particularly well-suited for applications where improved performance on reasoning and factual question-answering tasks, as reflected in its HellaSwag and TruthfulQA scores, is beneficial. Its fine-tuning on a specific dataset suggests potential for domain-specific applications aligned with the huangyt/FINETUNE2 dataset's content.