steven0226/llama-3.2-3b-taiwan-chat-clean
The steven0226/llama-3.2-3b-taiwan-chat-clean is a 3.2 billion parameter Llama-3.2-3B-Instruct model fine-tuned by steven0226 using QLoRA on a subset of the TaiwanChat dataset. This model is specifically optimized to enhance fluency and localization in Traditional Chinese, particularly for Taiwanese colloquialisms and cultural context. It is designed for research and non-commercial use, providing improved responses for Taiwan-specific queries compared to its base model.
Loading preview...
Overview
This model, steven0226/llama-3.2-3b-taiwan-chat-clean, is a 3.2 billion parameter Llama-3.2-3B-Instruct variant that has been fine-tuned using QLoRA (Unsloth) on a subset of the yentinglin/TaiwanChat dataset. The primary goal of this fine-tuning was to significantly improve the model's fluency and localization for Traditional Chinese, with a specific focus on Taiwanese language nuances and cultural context. The training involved 1 epoch on 14,633 filtered samples, with loss computed only on assistant responses.
Key Capabilities
- Enhanced Traditional Chinese Fluency: Demonstrates more natural and localized responses in Traditional Chinese, particularly for Taiwanese-specific queries.
- Taiwanese Context Understanding: Improved ability to handle questions related to Taiwanese culture, daily life, and common expressions, as shown in comparative examples.
- QLoRA Fine-tuning: Utilizes 4-bit NF4 loading with LoRA (r=16, alpha=16, dropout=0) across seven projection layers for efficient fine-tuning.
- Direct Use: Provided as a merged fp16 model, ready for direct loading with
transformers.
Limitations and Usage
- Factual Accuracy: May hallucinate or provide outdated information for factual queries (e.g., telecom rates, store hours).
- Language Mixing: Occasional presence of Simplified Chinese or English fragments due to training data composition.
- Safety Alignment: Safety behaviors are consistent with the base model, without additional safety alignment.
- License: Licensed under the Llama 3.2 Community License and Acceptable Use Policy. The training data (TaiwanChat) is CC BY-NC 4.0, making this model suitable only for research and non-commercial use.