seonglae/yokhal-md

Yokhal-md: A Korean Chatbot Based on Google Gemma

Yokhal-md is a 2.5 billion parameter language model developed by Seonglae Cho, fine-tuned from Google's Gemma-2b-it. This model is specifically designed for conversational interactions in Korean, with support for English, and incorporates elements of Korean internet culture.

Key Capabilities

• Bilingual Chatbot: Functions as a chatbot in both Korean and English.
• Cultural Nuance: Fine-tuned to understand and respond with Korean internet culture context.
• Gemma Architecture: Leverages the Gemma model's capabilities for efficient performance.
• Optimized for Chat: Preprocessing includes adapting system prompts for better chat interaction, placing them before user messages.

Training Details

The model underwent a multi-stage training process:

1. Weight Initialization: Started with weights initialized from an internet comments dataset.
2. Namuwiki Training: Further trained on the Korean Namuwiki dataset for 80,000 steps with a sequence length of 1024 and a batch size of 3 per device.
3. Instruction Tuning: Instruction-tuned on a Korean Instruction Dataset using QLoRa, with a sequence length of 2048.

Good For

• Korean-centric Chat Applications: Ideal for building chatbots that interact primarily in Korean.
• Cultural Context: Useful for applications requiring an understanding of Korean internet culture and colloquialisms.
• Research and Development: Provides a base for further fine-tuning or research into Korean language models.