Korean-OpenOrca-13B-v3 Overview

Developed by Kyujin Han (kyujinpy) as part of an LLM research consortium, Korean-OpenOrca-13B-v3 is a 13 billion parameter auto-regressive language model built upon the LLaMA2 transformer architecture. It utilizes hyunseoki/ko-en-llama2-13b as its base model.

Key Capabilities & Training

• Architecture: LLaMA2-based transformer, optimized for causal language modeling.
• Training Data: Fine-tuned using the OpenOrca-ko-v3 dataset, which is a DeepL translation of the original OpenOrca dataset.
• Performance: Achieves an average score of 48.86 across several Korean benchmarks, including Ko-ARC (43.77), Ko-HellaSwag (54.30), Ko-MMLU (41.79), Ko-TruthfulQA (43.85), and Ko-CommonGen V2 (60.57).
• Context Length: Supports a context length of 4096 tokens.

Use Cases

This model is particularly well-suited for:

• Korean Language Processing: Tasks requiring strong understanding and generation in Korean.
• Research and Development: As a base for further fine-tuning on specific Korean NLP applications.
• Benchmarking: Provides a solid baseline for evaluating other Korean LLMs.