weiren119/traditional_chinese_qlora_llama2_13b_merged
The weiren119/traditional_chinese_qlora_llama2_13b_merged model is a 13 billion parameter Llama 2-based language model, fine-tuned by weiren119 using QLoRA. It specializes in understanding and generating Traditional Chinese, specifically optimized for instruction-following tasks. This model leverages a translated Alpaca dataset to enhance its performance in Traditional Chinese contexts, making it suitable for applications requiring robust Traditional Chinese language capabilities.
Loading preview...
Traditional Chinese QLoRa Llama2 13b Merged Model
This model, developed by weiren119, is a 13 billion parameter Llama 2-based language model specifically fine-tuned for Traditional Chinese instruction-following tasks. It utilizes the QLoRA (Quantized Low-Rank Adaptation) method, enabling efficient fine-tuning on consumer-grade hardware (e.g., an RTX 3090 with 24GB VRAM).
Key Capabilities
- Traditional Chinese Proficiency: Optimized for understanding and generating text in Traditional Chinese.
- Instruction Following: Fine-tuned on a translated version of the Stanford Alpaca 52k dataset, enhancing its ability to follow instructions.
- Efficient Deployment: Merged QLoRA adapters allow for direct use of the fine-tuned model.
- Llama 2 Foundation: Benefits from the robust architecture and pre-training of the Llama 2 chat model.
Good For
- Applications requiring a strong understanding and generation of Traditional Chinese.
- Building chatbots or assistants that interact in Traditional Chinese.
- Research and development in Traditional Chinese NLP with a Llama 2 base.
- Scenarios where efficient, quantized models are preferred for deployment or resource constraints.
This model was trained using bitsandbytes 4-bit quantization with nf4 type and bfloat16 compute dtype, and PEFT 0.4.0. Resources for the 7B version and an online Colab demo are also available.