Overview

werty1248/Llama-3-Ko-8B-Instruct-AOG is an 8 billion parameter instruction-tuned model based on the Llama 3 architecture, specifically fine-tuned from beomi/Llama-3-Open-Ko-8B. This model is designed for enhanced instruction following in the Korean language.

Key Capabilities

• Korean Instruction Following: Optimized for generating detailed and friendly responses to user instructions in Korean.
• Dataset Integration: Trained on a combination of commercially available Korean datasets, including beomi/KoAlpaca-v1.1a, kyujinpy/OpenOrca-KO, and nlpai-lab/openassistant-guanaco-ko.
• Multi-turn Conversation: Incorporates multi-turn conversation data, though its reliability in this area is noted as lower.
• Performance: Shows an improvement in accuracy on the kobest_boolq benchmark (0.8312) compared to its base model (0.7963).

Training Details

The model was trained using Axolotl with LoRA for 3 epochs, utilizing a sequence length of 4096 and bf16 precision. Training took approximately 8 hours on a single A100 GPU.

Recommended Usage

It is recommended to use an "Instruction - Input - Response" or "Question - Instruction - Response" format for optimal performance, with the latter often yielding better results. The model aims to provide comprehensive answers without requiring external searches.