rinna/nekomata-7b
rinna/nekomata-7b is a 7 billion parameter causal language model developed by rinna, continually pre-trained from Qwen-7B on 30 billion tokens of mixed Japanese and English datasets. This model significantly enhances performance on Japanese tasks, leveraging Qwen's inclusive vocabulary (150k+) for efficient Japanese text processing. It supports a maximum context length of 32768 tokens, making it suitable for applications requiring extensive context and strong Japanese language capabilities.
Loading preview...
Overview
rinna/nekomata-7b is a 7 billion parameter language model developed by rinna, built upon the Qwen-7B architecture. It has undergone continual pre-training on approximately 30 billion tokens from a diverse mixture of Japanese and English datasets, including Japanese CC-100, Japanese C4, Japanese OSCAR, The Pile, and Wikipedia.
Key Capabilities & Features
- Enhanced Japanese Performance: The continual pre-training specifically targets and significantly improves the model's proficiency in Japanese language tasks.
- Efficient Japanese Tokenization: Utilizes the original Qwen tokenizer with a vocabulary size of 151,936, which offers superior tokenization efficiency for Japanese texts compared to models using the Llama-2 tokenizer, resulting in a lower byte-to-token rate.
- Extended Context Window: Supports a maximum sequence length of 32768 tokens, enabling the model to process and understand very long inputs.
- Robust Architecture: Based on a 32-layer, 4096-hidden-size transformer architecture, inheriting the robust design of the Qwen model family.
When to Use This Model
- Japanese Language Applications: Ideal for tasks requiring strong performance in Japanese, such as text generation, summarization, or translation.
- Long Context Processing: Suitable for use cases that benefit from a large context window, allowing for more comprehensive understanding and generation based on extensive input.
- Efficiency-Sensitive Tasks: Its optimized tokenization for Japanese can lead to more efficient processing and potentially lower computational costs for Japanese-centric applications.