Model Overview

CHIH-HUNG/llama-2-13b-Open_Platypus_and_ccp_2.6w-3_epoch is a 13 billion parameter language model built upon the meta-llama/Llama-2-13b-hf architecture. It has been fine-tuned using LoRA (rank 8) over 3 epochs, leveraging a combined dataset of approximately 25,000 entries from garage-bAInd/Open-Platypus and an additional 1,200 entries from the ccp dataset.

Key Capabilities & Performance

This model shows notable improvements in several benchmarks when compared to its base Llama-2-13b counterpart. Evaluation results from the HuggingFaceH4/open_llm_leaderboard indicate:

• HellaSwag: Achieves 82.56%, an increase from Llama-2-13b's 80.97%.
• TruthfulQA: Scores 42.09%, significantly higher than Llama-2-13b's 34.17%.
• MMLU: Shows a score of 55.84%, an improvement over Llama-2-13b's 54.34%.

The fine-tuning process utilized a single RTX4090 GPU, with a training loss of 0.6 over a runtime of approximately 12.5 hours using DeepSpeed.

Use Cases

This model is suitable for applications requiring enhanced general language understanding and generation, particularly where improved performance on reasoning and factual recall tasks, as indicated by the benchmark scores, is beneficial. Its fine-tuning on diverse datasets suggests a broad applicability for various NLP tasks.