surrey-nlp/diallm-llama-gspo-aus
The surrey-nlp/diallm-llama-gspo-aus model is an 8 billion parameter Llama 3.1-based language model developed by surrey-nlp, specifically adapted for Australian English (en-AU). It was continually pretrained on the International Corpus of English and fine-tuned using dialect-specific SFT on Multi-VALUE-transformed en-AU preference data, followed by Group Sequence Policy Optimization (GSPO). This model is designed to address the robustness-generation gap in English dialect adaptation, making it suitable for tasks requiring nuanced understanding and generation in Australian English.
Loading preview...
Overview
This model, surrey-nlp/diallm-llama-gspo-aus, is an 8 billion parameter variant of Llama 3.1, specifically engineered for Australian English (en-AU). It is a key component of the "DiaLLM: An Investigation into the Robustness-Generation Gap in English Dialect Adaptation" research, presented at EMNLP 2026.
Key Capabilities
- Dialect Adaptation: Explicitly adapted for Australian English, focusing on improving both robustness and generation quality for this specific dialect.
- Training Methodology: Utilizes a unique adaptation thread involving dialect-specific Supervised Fine-Tuning (SFT) on en-AU preference data, followed by Group Sequence Policy Optimization (GSPO).
- Base Model: Built upon the robust Llama 3.1-8B architecture, providing a strong foundation for dialectal nuances.
- Continual Pretraining: Underwent continual pretraining on the International Corpus of English, encompassing 18 varieties and approximately 20 million tokens.
Good For
- Applications requiring high-quality text generation and understanding in Australian English.
- Research into dialectal variations and their impact on large language models.
- Developers interested in models fine-tuned with GSPO for specific linguistic targets.
For more technical details, including code, checkpoints, and preference datasets, refer to the official GitHub repository and the research paper.