Hugging Face Overview
shenzhi-wang/Llama3.1-70B-Chinese-Chat is a 70 billion parameter instruction-tuned language model developed by Shenzhi Wang and Yaowei Zheng, built upon the Meta-Llama-3.1-70B-Instruct base model. It is specifically fine-tuned for both Chinese and English users, enhancing its utility in bilingual environments. The model utilizes the ORPO fine-tuning algorithm, a reference-free monolithic preference optimization method, to achieve its specialized capabilities.
Key Capabilities
- Bilingual Support: Optimized for both Chinese and English users.
- Enhanced Roleplay: Demonstrates significant improvements in roleplaying scenarios.
- Function Calling: Features enhanced capabilities for function calling, useful for tool-use applications.
- Mathematical Proficiency: Exhibits improved performance in mathematical tasks.
- Context Length: Supports a context length of 32,768 tokens, inherited from its base model.
Training Details
The model was fine-tuned over 3 epochs with a learning rate of 1.5e-6 and a cosine learning rate scheduler. It used a cutoff length of 8192 and an ORPO beta of 0.05, with full parameter fine-tuning and a paged_adamw_32bit optimizer. The training dataset included over 100,000 preference pairs.
Good for
- Applications requiring strong bilingual (Chinese/English) conversational abilities.
- Use cases involving complex roleplaying or character interactions.
- Scenarios where robust function calling and tool integration are necessary.
- Tasks demanding improved mathematical reasoning and problem-solving.