Taiwan-tinyllama-v1.1-base: Traditional Chinese Causal Language Model

This model, developed by benchang1110, is a continue-pretrained version of TinyLlama-v1.1 specifically optimized for traditional Chinese. It has undergone further training on a substantial dataset exceeding 10 billion tokens of traditional Chinese content.

Key Characteristics & Performance

• Language Specialization: Primarily designed for generating text in traditional Chinese.
• Efficiency: Utilizes bfloat16 for inference, requiring only around 3GB of VRAM, making it accessible for environments with limited resources.
• Training Details: The model was trained for 1 epoch with a global batch size of 32, a learning rate of 5e-5, and a max sequence length of 2048. The training consumed approximately 200 GPU hours on a single A100 (80GB) GPU.

Intended Use

• Causal Language Modeling: This model functions as a causal language model, meaning it generates text based on preceding text. It is not designed as a chat model or for generating human-like conversational responses.
• Text Generation: Ideal for tasks requiring the generation of traditional Chinese text, such as content creation, data augmentation, or research in traditional Chinese NLP.

Usage Notes

Developers can easily load and use the model with the transformers library, specifying attn_implementation="flash_attention_2" and torch_dtype=torch.bfloat16 for optimal performance and memory usage.