ananddey/gemma-4-E2B-asm-init

Overview

This model, ananddey/gemma-4-E2B-asm-init, is a pre-initialized checkpoint based on the google/gemma-4-E2B architecture, specifically prepared for continued pretraining (CPT) on Assamese text. It is not a fully trained or instruction-tuned model, but rather a foundation for building a dedicated Assamese language model.

Key Differentiators

• Assamese-Optimized Tokenizer: Replaces the original Gemma tokenizer with a custom 32K SentencePiece BPE tokenizer trained on the AsmCorpus dataset, specifically for Assamese.
• FOCUS-Aligned Embeddings: The embedding table is initialized using FOCUS (Fast Overlapping token Combinations Using Surface-form), which semantically aligns new Assamese tokens with the original Gemma embeddings. This bypasses the need for a lengthy embedding warmup phase during CPT.
• Unchanged Transformer Weights: The core transformer layers from the base google/gemma-4-E2B model remain untouched, providing a strong foundation for continued training.

Intended Use

This model is ideal for:

• Continued Pretraining (CPT): Serving as a ready-to-use starting point for training a large language model on Assamese corpora.
• Assamese NLP Research: Providing a specialized base for developing and experimenting with Assamese-specific NLP applications.

Usage Notes

• The model requires torch_dtype=torch.bfloat16 and attn_implementation="sdpa" for proper loading due to Gemma 4's global layers.
• It is not instruction-tuned or chat-aligned and should not be used as an end-user assistant without further training.