Parum-Lucis/gemma3-270m-bpe-4K-20y-merged
Parum-Lucis/gemma3-270m-bpe-4K-20y-merged is a 0.3 billion parameter language model based on the Google Gemma-3 architecture. This model is a merge of pre-trained language models, specifically combining google/gemma-3-270m with rafurafu/gemma-3-270m-cpt-bpe-4K-20y using the Task Arithmetic method. It is designed for general language understanding and generation tasks, leveraging its compact size for efficient deployment.
Loading preview...
Model Overview
Parum-Lucis/gemma3-270m-bpe-4K-20y-merged is a compact 0.3 billion parameter language model built upon the Google Gemma-3 architecture. This model was created using the mergekit tool, specifically employing the Task Arithmetic method to combine different pre-trained models.
Merge Details
The base model for this merge was google/gemma-3-270m. It was merged with rafurafu/gemma-3-270m-cpt-bpe-4K-20y. The merging process involved combining specific layers (0 to 18) from both models, with a weight of 1.0 applied to the rafurafu model's contribution within these layers. The tokenizer configuration was sourced from the base model, and the merge was performed using bfloat16 data type.
Key Characteristics
- Architecture: Based on the Google Gemma-3 family.
- Parameter Count: 0.3 billion parameters, making it suitable for resource-constrained environments.
- Merge Method: Utilizes Task Arithmetic for combining model weights, potentially enhancing specific capabilities or robustness.
- Context Length: Supports a context length of 32768 tokens.
Potential Use Cases
This model is suitable for applications requiring a small, efficient language model, such as:
- Text generation in low-resource settings.
- Experimentation with merged model architectures.
- Fine-tuning for specific downstream tasks where a compact model is advantageous.