Hugging Face Overview
Saxo/Linkbricks-Horizon-AI-Llama-3.3-Korean-70B-sft-dpo Overview
This model is a 70 billion parameter Korean-enhanced language model developed by Yunsung Ji (Saxo), a data scientist at Linkbricks. It is built upon the meta-llama/Llama-3.3-70B-Instruct base model and has undergone Supervised Fine-Tuning (SFT) followed by Direct Preference Optimization (DPO) using 8 H100-80G GPUs.
Key Capabilities & Training
- Multilingual Enhancement: Trained on 40 million Korean news and wiki corpus, augmented with cross-lingual data for Korean, Japanese, Chinese, and English, enabling robust performance across these languages.
- Advanced Reasoning: Incorporates mathematical and logical judgment data to handle complex logical problems effectively.
- Context Window: Supports a substantial 32,768 token context window.
- Functionality: Enhanced with Function Calling and Tool Calling capabilities.
- Tokenizer: Utilizes the base model's tokenizer without word expansion.
- Training Methodology: Employs Deepspeed Stage 3, rslora, and BAdam Layer Mode for efficient training.
Ideal Use Cases
This model is particularly well-suited for applications requiring:
- High-dimensional analysis of customer reviews and social media posts.
- Code generation and programming tasks.
- Creative and technical writing.
- Solving mathematical problems.
- Complex logical reasoning and decision-making processes.