steven0226/llama-3.2-3b-taiwan-chat

Overview

steven0226/llama-3.2-3b-taiwan-chat is a Llama-3.2-3B-Instruct model that has been fine-tuned using QLoRA (Unsloth) on a subset of the yentinglin/TaiwanChat dataset. The primary goal of this fine-tuning was to enhance the model's fluency and localization for Traditional Chinese, specifically with a Taiwanese linguistic style. The model's weights have been merged into a full fp16 version, making it ready for direct use with libraries like Hugging Face Transformers.

Key Features & Training Details

• Base Model: unsloth/Llama-3.2-3B-Instruct (a mirror of meta-llama/Llama-3.2-3B-Instruct).
• Fine-tuning Method: QLoRA with 4-bit NF4 loading and LoRA (r=16, alpha=16, dropout=0) applied to seven projection layers.
• Dataset: A sampled subset of 14,776 entries from yentinglin/TaiwanChat, filtered for sequences under 2048 tokens.
• Context Length: max_seq_length set to 2048 tokens.
• Training: 1 epoch, with loss computed only on assistant responses (train_on_responses_only).
• Performance: Demonstrates improved responses for Taiwanese-specific queries compared to the base model, as shown in provided examples covering topics like bubble tea, phone plans, Tainan night markets, and cross-strait terminology.

Limitations

• May produce outdated or hallucinated factual content (e.g., telecom rates, store information).
• Occasional leakage of Simplified Chinese or English fragments due to training data composition.
• No additional safety alignment beyond the base model's behavior.

Usage & Licensing

This model is distributed under the Llama 3.2 Community License and its Acceptable Use Policy. The training data, TaiwanChat, is licensed under CC BY-NC 4.0, meaning this model is for research/non-commercial use only.