Model Overview
CHIH-HUNG/llama-2-13b-Open_Platypus_and_ccp_2.6w is a 13 billion parameter model built upon the Llama-2-13b architecture. It has been fine-tuned using LoRA (rank 8) on a combination of the garage-bAInd/Open-Platypus dataset (approximately 25,000 entries) and an additional CCP dataset (approximately 1,200 entries), totaling around 26,200 data points. The training was conducted for one epoch with a learning rate of 5e-5, utilizing bf16 precision and 4-bit quantization.
Performance Benchmarks
Evaluation against the HuggingFaceH4/open_llm_leaderboard shows that this model generally outperforms the base meta-llama/Llama-2-13b-hf across several benchmarks, including ARC, HellaSwag, MMLU, and TruthfulQA. While its average score is 59.41, it specifically shows an improvement in HellaSwag (82.51 vs 80.97) and MMLU (56.12 vs 54.34) compared to the base Llama-2-13b.
Key Characteristics
- Base Model: Llama-2-13b
- Fine-tuning Datasets: Open-Platypus and CCP
- PEFT Method: LoRA with
gate_proj, up_proj, down_proj targets - Context Length: 4096 tokens
Use Cases
This model is suitable for applications requiring a capable instruction-following LLM, particularly where improved performance over the base Llama-2-13b on reasoning and common sense tasks is beneficial. Its training on diverse instruction datasets suggests applicability in general conversational agents and task automation.