sajalregmi4/arkios-1b-base
Arkios-1B-base by sajalregmi4 is a 1.04 billion-parameter bilingual English-Nepali base model, pretrained from scratch on 150 billion tokens. It features a custom tokenizer specifically designed for Devanagari script, achieving 1.69 tokens/word on Nepali text, which addresses a common limitation in other bilingual models. This model is intended as a fine-tuning foundation or for completion/autocomplete tasks, with a Qwen3-compatible dense transformer architecture and a 4096-token context length.
Loading preview...
Arkios-1B-base: A Bilingual English-Nepali Foundation Model
Arkios-1B-base is a 1.04 billion-parameter model developed by sajalregmi4, pretrained from scratch on 150 billion tokens. Unlike many bilingual models, it features a purpose-built tokenizer for Devanagari script, which correctly handles Nepali vowel signs and virama, leading to significantly better tokenization efficiency (1.69 tokens/word on Nepali). This addresses a structural limitation found in models using standard Latin-script tokenizers for Devanagari.
Key Capabilities & Features
- Bilingual Proficiency: Designed for both English and Nepali, with a focus on improving Nepali language processing through its specialized tokenizer.
- Base Model: Serves as a foundation for fine-tuning or for direct use in completion and autocomplete scenarios. An instruction-tuned version,
arkios-1b-chat, is also available. - Robust Architecture: Utilizes a Qwen3-compatible dense transformer architecture, incorporating GQA, RMSNorm, SwiGLU, RoPE, and QK-norm, with a context length of 4096 tokens.
- Extensive Pretraining: Trained on a diverse dataset including FineWeb-Edu, OpenWebMath, GitHub-code-clean, and various Nepali corpora, totaling 150 billion tokens.
- Evaluation Insights: While scoring at chance in standard multiple-choice letter formats for both languages, the model demonstrates real Nepali comprehension (0.306 accuracy) when evaluated on direct answer text, indicating its ability to understand the language despite prompt-following limitations in its base form.
Good For
- Fine-tuning: Ideal as a base model for developing custom instruction-tuned or task-specific models for English and Nepali applications.
- Completion & Autocomplete: Suitable for generating text completions or providing autocomplete suggestions in both languages.
- Nepali Language Applications: Particularly strong for use cases requiring accurate and efficient processing of Nepali text, thanks to its specialized tokenizer.