shisa-ai/ablation-178-finalsft2-shisa-v2-mistral-nemo-japanese-12b

Overview

This model, ablation-178-finalsft2-shisa-v2-mistral-nemo-japanese-12b, is a 12.2 billion parameter language model developed by shisa-ai. It is built upon the cyberagent/Mistral-Nemo-Japanese-Instruct-2408 base model and has undergone extensive supervised fine-tuning (SFT) using a diverse collection of Japanese-centric datasets. The training leveraged the Axolotl framework, incorporating specific architectural modifications via the Liger plugin, such as liger_rope, liger_rms_norm, and liger_glu_activation, which may contribute to its performance characteristics.

Key Capabilities

• Enhanced Japanese Language Understanding: Fine-tuned on multiple Japanese datasets, including shisa-v2-best-of-n-athenev2-tulu70b-llama33-only-no-sysprompt, shisa-v2-roleplaying-sft, translation_set_april_6, rewild-set-deepseek-subset, magpie-ultra-set, magpie-advanced-questions-set, and japan-magpie-set.
• Instruction Following: Specifically trained on shisa-v2-instruction-following-sft to improve adherence to user instructions.
• Roleplaying and Conversational Abilities: Benefits from fine-tuning on roleplaying datasets, suggesting improved performance in dialogue-based applications.
• Optimized Training: Utilizes flash_attention and gradient_checkpointing for efficient training, with a sequence length of 8192 tokens.

Good for

• Applications requiring robust Japanese language generation and comprehension.
• Instruction-following tasks in Japanese.
• Developing conversational AI and roleplaying agents for Japanese users.
• Research into the effects of specific architectural modifications (Liger plugin) on Japanese LLM performance.