NYTK/PULI-Trio-Q
NYTK/PULI-Trio-Q is a 7.62 billion parameter language model, continually pretrained from Qwen2.5 7B Instruct. Developed by NYTK, this model is specifically enhanced for Hungarian language processing, incorporating a substantial Hungarian dataset alongside English and Chinese data. It features a notable maximum sequence length of 32,768 tokens, making it suitable for applications requiring extensive context understanding in multilingual environments, particularly for Hungarian-centric tasks.
Loading preview...
Model Overview
NYTK/PULI-Trio-Q is a 7.62 billion parameter language model, continually pretrained from the Qwen2.5 7B Instruct architecture. This model has been specifically enhanced for multilingual capabilities, with a strong focus on the Hungarian language, making it a valuable resource for applications requiring robust Hungarian language understanding and generation.
Key Capabilities & Training
The model's training regimen involved a diverse dataset, including:
- Hungarian Data: 8.7 billion words from documents exceeding 5000 words, Hungarian Wikipedia, and news articles. An additional 626 million Hungarian words (1 epoch) from Wikipedia and news were used for specific training.
- English Data: Long Context QA (1 billion words) and BookSum (42 million words).
- Chinese Data: 3 billion Chinese characters from the Wudao dataset.
Notable Features
- Extended Context Length: Supports a
max_seq_lengthof 32,768 tokens, enabling the processing of very long texts. - Multilingual Focus: While building on a strong base, its continued pretraining significantly boosts its performance and utility for Hungarian language tasks.
Usage
This model can be easily integrated into projects using the Hugging Face transformers library, as demonstrated by the provided pipeline example for text generation. For academic or research use, please cite the associated ChatPULI paper.