Model Overview

logicker/SkkuDS-DPO-72B-v3 is a 72.3 billion parameter language model based on the Qwen1.5 architecture, a transformer-based decoder-only model. It has been fine-tuned using Direct Preference Optimization (DPO) on the Intel/orca_dpo_pairs dataset, enhancing its ability to follow instructions and generate high-quality responses.

Key Capabilities

• Large Scale: With 72.3 billion parameters, it offers robust language understanding and generation.
• Extended Context: Provides stable support for a 32K token context length, suitable for processing longer inputs and generating comprehensive outputs.
• Multilingual Support: Features improved multilingual capabilities for both base and chat model applications.
• DPO Fine-tuning: Leverages DPO on a high-quality dataset for better alignment with human preferences and instruction following.
• Architectural Enhancements: Built on the Transformer architecture with SwiGLU activation, attention QKV bias, and group query attention, though GQA and mixed attention were not included in this beta version.

Good For

• Applications requiring advanced natural language processing.
• Tasks benefiting from a large context window.
• Scenarios where instruction-following and preference alignment are critical.
• Multilingual text generation and understanding.