damjanzimbakov/qwen3-1.7b-macedonian-pretrain
The damjanzimbakov/qwen3-1.7b-macedonian-pretrain model is a 1.7 billion parameter Qwen3-based causal language model developed by damjanzimbakov. It has undergone continued pretraining specifically on a Macedonian text corpus for 3 epochs. This model is optimized for generating text in Macedonian, making it suitable for applications requiring Macedonian language understanding and generation.
Loading preview...
Overview
This model, damjanzimbakov/qwen3-1.7b-macedonian-pretrain, is a 1.7 billion parameter variant of the Qwen3-1.7B base model that has been further pretrained on Macedonian text. Its primary purpose is causal language modeling, focusing on generating coherent text in Macedonian.
Key Characteristics
- Base Model: Qwen/Qwen3-1.7B-Base architecture.
- Continued Pretraining: Specialized pretraining on a dedicated Macedonian corpus (
LVSTCK/macedonian-corpus-cleaned-dedup). - Training Details: Trained for 3 epochs using
bfloat16precision andflash_attention_2with a sequence length of2048. - Limitations: As a pretrained model, it is not instruction-tuned and may produce incomplete or noisy outputs. It is designed for foundational language understanding rather than direct conversational use.
Use Cases
This model is particularly well-suited for tasks requiring a strong understanding and generation capability in the Macedonian language. Potential applications include:
- Macedonian text generation: Creating content, summaries, or creative writing in Macedonian.
- Language research: Studying Macedonian language patterns and structures.
- Foundation for fine-tuning: Serving as a robust base model for further instruction-tuning or task-specific fine-tuning on Macedonian datasets.