Model Overview

This model, CHIH-HUNG/llama-2-13b-OpenOrca_20w, is a 13 billion parameter language model fine-tuned by CHIH-HUNG. It is based on the meta-llama/Llama-2-13b-hf architecture and was trained using a subset of the OpenOrca dataset, specifically the first 200,000 training examples.

Fine-Tuning Details

The fine-tuning process utilized LoRA (Low-Rank Adaptation) with a rank of 8, targeting q_proj and v_proj layers. Training was conducted for 1 epoch with a learning rate of 5e-5, using bf16 precision and load_in_4bit quantization. The training loss achieved was 0.8616 over approximately 29 hours.

Evaluation and Performance

Evaluations were performed against four benchmarks from the HuggingFaceH4/open_llm_leaderboard: ARC, HellaSwag, MMLU, and TruthfulQA. The model achieved an average score of 60.46, with specific scores of 59.9 on ARC, 82.51 on HellaSwag, 56.3 on MMLU, and 43.14 on TruthfulQA. These results indicate its capability in general reasoning and question-answering tasks, positioning it competitively among other Llama-2-13b variants fine-tuned on OpenOrca.

Use Cases

This model is well-suited for applications requiring a capable 13B parameter model with enhanced instruction-following abilities, particularly in areas where the OpenOrca dataset's diverse instruction-tuning data is beneficial. It can be used for conversational agents, question answering, and general text generation tasks.