Taiwan LLM 7B v2.1 Chat: Tailored for Traditional Chinese

Developed by Yen-Ting Lin and Yun-Nung Chen, the Taiwan LLM 7B v2.1 chat is a 7 billion parameter GPT-like model specifically fine-tuned for Traditional Chinese. It builds upon the yentinglin/Taiwan-LLM-7B-v2.0-base model and incorporates diverse Taiwanese textual sources, refined through Supervised Fine-Tuning.

Key Capabilities

• Culturally Aligned Language Understanding: Excels in comprehending and generating Traditional Chinese text with a strong alignment to Taiwan's linguistic and cultural contexts.
• Improved Benchmark Performance: Demonstrates enhanced performance on benchmarks such as TC-Eval, indicating strong contextual comprehension.
• Chat-Optimized: Designed for conversational applications, providing helpful, detailed, and polite responses.

Training Details

The model was fine-tuned using a mix of publicly available and synthetic datasets. Key training hyperparameters included a learning rate of 5e-05, Adam optimizer with betas=(0.9,0.999) and epsilon=1e-08, and a cosine learning rate scheduler with a warmup ratio of 0.03 over 5 epochs.

Intended Uses

This model is ideal for applications requiring nuanced understanding and generation of Traditional Chinese, particularly within a Taiwanese cultural context. It can be integrated using the pipeline() function from Hugging Face Transformers for text generation tasks.