Overview

OPI-PG/Qra-13b is a 13 billion parameter foundational language model, a collaborative effort between the National Information Processing Institute (OPI) and Gdańsk University of Technology (PG). It was initialized with Llama 2 13B weights and then extensively trained on a meticulously cleaned, filtered, and deduplicated corpus of approximately 90 billion Polish tokens, primarily sourced from web data like CommonCrawl and MADLAD-400. The model was trained for one epoch on sequences of 4096 tokens, utilizing modern optimizations such as Flash Attention 2 and FSDP.

Key Capabilities

• Polish Language Specialization: Specifically adapted and trained on a vast Polish text corpus, making it highly proficient in Polish language understanding and generation.
• Foundation Model: Designed as a base model for further fine-tuning, it is not intended for direct conversational or instruction-following tasks without additional adaptation.
• Strong Perplexity Performance: Achieves a perplexity of 10.5 on the PolEval-2018 dataset and 4.2 on a new 2024 long document dataset, outperforming other Polish and English models in its class on Polish texts.

Good For

• Developing Polish-centric LLM applications: Ideal for researchers and developers building applications that require deep understanding and generation of Polish text.
• Further Fine-tuning: Serves as an excellent base for fine-tuning into instruction-following, conversational, or task-specific models for the Polish language.
• Research in Polish NLP: Useful for academic and industrial research into large language models tailored for less-resourced languages or specific linguistic challenges in Polish.