EdonFetaji/MK-Llama-3.2-1B
EdonFetaji/MK-Llama-3.2-1B is a 1 billion parameter Llama-3.2 model that has undergone continued pretraining specifically for the Macedonian language. This model was trained using LoRA adapters on the lvstck/macedonian-corpus-cleaned-dedup dataset. It is optimized for tasks requiring proficiency in Macedonian, making it suitable for applications targeting this specific language.
Loading preview...
Overview
EdonFetaji/MK-Llama-3.2-1B is a 1 billion parameter language model based on the Llama-3.2 architecture. Its primary distinction is its continued pretraining for the Macedonian language, utilizing the lvstck/macedonian-corpus-cleaned-dedup dataset. This specialized training aims to enhance its performance and understanding of Macedonian.
Training Details
The model underwent a two-stage continued pretraining process, totaling 1.3 epochs. The training was conducted using LoRA adapters on a single A100 GPU, with stages split between Google Colab and a FINKI GPU cluster. Training metrics, including loss, learning rate, and gradient norm, are available for Stage 2 via a TensorBoard dashboard.
Key Capabilities
- Macedonian Language Proficiency: Specialized for tasks and applications requiring strong performance in Macedonian.
- Efficient Training: Utilizes LoRA adapters, enabling efficient continued pretraining on single-GPU setups.
Use Cases
This model is particularly well-suited for:
- Macedonian NLP applications: Such as text generation, translation, or analysis in Macedonian.
- Research and development: For exploring language models tailored to specific, less-resourced languages.