The logicker/SkkuDS-DPO-72B-v3 is a 72.3 billion parameter Qwen1.5-based decoder-only language model, fine-tuned using DPO on the Intel/orca_dpo_pairs dataset. This model offers stable support for a 32K context length and enhanced multilingual capabilities. It is designed for advanced natural language understanding and generation tasks, leveraging its large parameter count and DPO optimization for improved instruction following.
Loading preview...
Model Overview
logicker/SkkuDS-DPO-72B-v3 is a 72.3 billion parameter language model based on the Qwen1.5 architecture, a transformer-based decoder-only model. It has been fine-tuned using Direct Preference Optimization (DPO) on the Intel/orca_dpo_pairs dataset, enhancing its ability to follow instructions and generate high-quality responses.
Key Capabilities
- Large Scale: With 72.3 billion parameters, it offers robust language understanding and generation.
- Extended Context: Provides stable support for a 32K token context length, suitable for processing longer inputs and generating comprehensive outputs.
- Multilingual Support: Features improved multilingual capabilities for both base and chat model applications.
- DPO Fine-tuning: Leverages DPO on a high-quality dataset for better alignment with human preferences and instruction following.
- Architectural Enhancements: Built on the Transformer architecture with SwiGLU activation, attention QKV bias, and group query attention, though GQA and mixed attention were not included in this beta version.
Good For
- Applications requiring advanced natural language processing.
- Tasks benefiting from a large context window.
- Scenarios where instruction-following and preference alignment are critical.
- Multilingual text generation and understanding.