KBlueLeaf/guanaco-7b-leh-v2
KBlueLeaf/guanaco-7b-leh-v2 is a 7 billion parameter multilingual instruction-following language model based on the LLaMA architecture, developed by KohakuBlueleaf. It is fine-tuned to improve performance in Chinese and Japanese, making it suitable for instruction-based prompts and chatbot applications. The model features an increased context length of 4096 tokens and is trained on a larger dataset including alpaca-cleaned and GuanacoDataset.
Loading preview...
Overview
KBlueLeaf/guanaco-7b-leh-v2 is a 7 billion parameter LLaMA-based instruction-following model developed by KohakuBlueleaf. It is specifically fine-tuned using guanaco-lora with LoRA, embed_tokens, and lm_head training to enhance its multilingual capabilities, particularly in Chinese and Japanese. This model is designed for instruction-based interactions and chatbot usage.
Key Enhancements & Features
- Multilingual Performance: Improved handling of Chinese and Japanese compared to the original LLaMA, achieved by training
embed_tokensandlm_head. - Increased Context Length: The context cutoff length has been expanded to 4096 tokens.
- Larger Training Dataset: Trained on a combined dataset of
alpaca-cleanedandGuanacoDataset, totaling 540,000 entries. - Chatbot Optimization: Enhanced for chat-based interactions due to a larger proportion of chat-based data in its training set.
- Training Methodology: Utilizes bf16 training, a larger batch size (128), and a specific LoRA approach for attention parts to mitigate overfitting/memorization issues often seen in native finetuning.
Recommended Use Cases
- Instruction Following: Excels at responding to explicit instructions.
- Chatbot Applications: Well-suited for conversational AI due to its training on chat-based data.
- Multilingual Tasks: Particularly effective for tasks involving Chinese and Japanese language processing.