timothydillan/gemma4-e2b-balinese-cpt
timothydillan/gemma4-e2b-balinese-cpt is a 5.1 billion parameter Gemma-based causal language model developed by timothydillan. It has undergone continued pretraining on a Balinese text corpus of approximately 25 million tokens, enhancing its fluency in the Balinese language. This model serves as a base/fluency model, specifically designed to build Balinese language proficiency before subsequent instruction-tuning. It is intended for research use, particularly for developing Balinese language assistants.
Loading preview...
Overview
This model, timothydillan/gemma4-e2b-balinese-cpt, is the first stage in the development of a Balinese assistant under the Open Indonesia Models initiative. It is a 5.1 billion parameter Gemma-based model that has undergone continued pretraining (CPT) on a substantial Balinese text corpus, comprising approximately 25 million tokens. The primary goal of this CPT phase is to establish strong Balinese language fluency.
Key Characteristics
- Base/Fluency Model: This is a foundational model focused on language proficiency, not an instruction-following assistant. It is designed to be the base upon which instruction-tuned assistants are built.
- Balinese Language Focus: Specifically trained to enhance understanding and generation of the Balinese language.
- Gemma Architecture: Built upon the
google/gemma-4-E2B-itmodel. - Merged Weights: Provided with merged 16-bit weights, allowing direct loading with
transformersorUnsloth. - LoRA Adapter Included: The LoRA adapter from the CPT run is included for reference.
Usage and Licensing
This model can be loaded using the AutoModelForCausalLM and AutoTokenizer classes from the transformers library. The base Gemma weights are under the Gemma license, and the continued-pretrained Balinese corpora are intended for research use.