CHIH-HUNG/llama-2-13b-FINETUNE1_17w-r16
CHIH-HUNG/llama-2-13b-FINETUNE1_17w-r16 is a 13 billion parameter Llama-2-based language model fine-tuned by CHIH-HUNG. It was trained using the huangyt/FINETUNE1 dataset, comprising approximately 170,000 data points, with a LoRA rank of 16. This model demonstrates improved performance on benchmarks like MMLU and TruthfulQA compared to the base Llama-2-13b model, making it suitable for general language understanding and generation tasks.
Loading preview...
Model Overview
This model, CHIH-HUNG/llama-2-13b-FINETUNE1_17w-r16, is a 13 billion parameter language model built upon the meta-llama/Llama-2-13b-hf architecture. It has been fine-tuned by CHIH-HUNG using the huangyt/FINETUNE1 dataset, which consists of approximately 170,000 training examples.
Fine-Tuning Details
The fine-tuning process utilized LoRA (Low-Rank Adaptation) with a rank of 16, targeting the gate_proj, up_proj, and down_proj layers. Training was conducted on a single RTX4090 GPU with bf16 precision and 4-bit quantization, achieving a train_loss of 0.66 over one epoch.
Performance Benchmarks
Evaluations against the HuggingFaceH4/open_llm_leaderboard show that this fine-tuned model generally outperforms the base meta-llama/Llama-2-13b-hf across several benchmarks, including ARC, HellaSwag, MMLU, and TruthfulQA. Notably, it achieved an average score of 58.86, with specific improvements in MMLU (56.16) and TruthfulQA (39.75) compared to the base model's 54.34 and 34.17 respectively.
Potential Use Cases
- General Language Tasks: Improved performance on common benchmarks suggests suitability for a wide range of natural language understanding and generation applications.
- Research and Development: Provides a fine-tuned Llama-2 variant for further experimentation and specialized task adaptation.