tokyotech-llm/Swallow-70b-NVE-instruct-hf
The Swallow-70b-NVE-instruct-hf model by tokyotech-llm is a 69 billion parameter instruction-tuned language model continually pre-trained from the Llama 2 family, with a focus on Japanese language data. This model utilizes a tokenizer without vocabulary expansion (NVE) and is optimized for efficient Japanese text processing. It demonstrates strong performance across various Japanese language tasks, making it suitable for applications requiring robust Japanese language understanding and generation.
Loading preview...
Overview
The tokyotech-llm/Swallow-70b-NVE-instruct-hf is a 69 billion parameter instruction-tuned language model developed by tokyotech-llm. It is built upon the Llama 2 architecture and has undergone continual pre-training with a significant addition of Japanese language data. This specific variant, NVE (No Vocabulary Expansion), uses a tokenizer that does not expand its vocabulary, which can be beneficial for certain applications.
Key Capabilities
- Enhanced Japanese Language Performance: The model shows substantial improvements over the base Llama 2 70B model on various Japanese benchmarks, including JCommonsenseQA, JEMHopQA, NIILC, JSQuAD, XL-Sum, MGSM, and WMT20-en-ja.
- Instruction Following: As an instruction-tuned model, it is designed to understand and execute commands provided in natural language.
- Efficient Tokenization: The NVE tokenizer aims for efficient text representation, potentially leading to faster inference for Japanese content.
Training Details
The model was continually pre-trained using a diverse set of datasets, including Japanese Wikipedia, RefinedWeb, Swallow Corpus, and The Pile. Instruction tuning was performed using Japanese-translated versions of Anthropic HH-RLHF, Databricks Dolly 15-k, and OpenAssistant Conversations Dataset.
Good For
- Applications requiring strong Japanese language understanding and generation.
- Tasks such as question answering, summarization, and machine translation in Japanese.
- Developers looking for a large-scale, instruction-tuned model with a focus on Japanese linguistic capabilities.