benchang1110/Taiwan-tinyllama-v1.1-base
The benchang1110/Taiwan-tinyllama-v1.1-base is a continue-pretrained version of TinyLlama-v1.1, specifically tailored for traditional Chinese. This causal language model was trained on over 10 billion tokens of traditional Chinese data, making it highly specialized for text generation in this language. It requires approximately 3GB of VRAM for inference using bfloat16, offering an efficient solution for traditional Chinese language processing tasks.
Loading preview...
Taiwan-tinyllama-v1.1-base: Traditional Chinese Causal Language Model
This model, developed by benchang1110, is a continue-pretrained version of TinyLlama-v1.1 specifically optimized for traditional Chinese. It has undergone further training on a substantial dataset exceeding 10 billion tokens of traditional Chinese content.
Key Characteristics & Performance
- Language Specialization: Primarily designed for generating text in traditional Chinese.
- Efficiency: Utilizes
bfloat16for inference, requiring only around 3GB of VRAM, making it accessible for environments with limited resources. - Training Details: The model was trained for 1 epoch with a global batch size of 32, a learning rate of 5e-5, and a max sequence length of 2048. The training consumed approximately 200 GPU hours on a single A100 (80GB) GPU.
Intended Use
- Causal Language Modeling: This model functions as a causal language model, meaning it generates text based on preceding text. It is not designed as a chat model or for generating human-like conversational responses.
- Text Generation: Ideal for tasks requiring the generation of traditional Chinese text, such as content creation, data augmentation, or research in traditional Chinese NLP.
Usage Notes
Developers can easily load and use the model with the transformers library, specifying attn_implementation="flash_attention_2" and torch_dtype=torch.bfloat16 for optimal performance and memory usage.