Model Overview

This model, CHIH-HUNG/llama-2-13b-FINETUNE1_17w-r16, is a 13 billion parameter language model built upon the meta-llama/Llama-2-13b-hf architecture. It has been fine-tuned by CHIH-HUNG using the huangyt/FINETUNE1 dataset, which consists of approximately 170,000 training examples.

Fine-Tuning Details

The fine-tuning process utilized LoRA (Low-Rank Adaptation) with a rank of 16, targeting the gate_proj, up_proj, and down_proj layers. Training was conducted on a single RTX4090 GPU with bf16 precision and 4-bit quantization, achieving a train_loss of 0.66 over one epoch.

Performance Benchmarks

Evaluations against the HuggingFaceH4/open_llm_leaderboard show that this fine-tuned model generally outperforms the base meta-llama/Llama-2-13b-hf across several benchmarks, including ARC, HellaSwag, MMLU, and TruthfulQA. Notably, it achieved an average score of 58.86, with specific improvements in MMLU (56.16) and TruthfulQA (39.75) compared to the base model's 54.34 and 34.17 respectively.

Potential Use Cases

• General Language Tasks: Improved performance on common benchmarks suggests suitability for a wide range of natural language understanding and generation applications.
• Research and Development: Provides a fine-tuned Llama-2 variant for further experimentation and specialized task adaptation.