sathiiiii/polyalign-gemma2-2b-en-dist-sft
The sathiiiii/polyalign-gemma2-2b-en-dist-sft model is a 2.6 billion parameter language model, fine-tuned from Google's Gemma-2-2B architecture. This model has been specifically fine-tuned on the polyalign_dist_sft_train dataset, indicating an optimization for specific supervised fine-tuning tasks. It is designed for applications requiring a compact yet capable model within the Gemma 2 family.
Loading preview...
Model Overview
The sathiiiii/polyalign-gemma2-2b-en-dist-sft is a 2.6 billion parameter language model derived from the google/gemma-2-2b base architecture. This model has undergone supervised fine-tuning (SFT) using the polyalign_dist_sft_train dataset, suggesting a specialization for tasks aligned with this training data.
Key Training Details
- Base Model:
google/gemma-2-2b - Fine-tuning Dataset:
polyalign_dist_sft_train - Parameters: 2.6 billion
- Context Length: 8192 tokens
- Learning Rate: 1e-05
- Optimizer: ADAMW_TORCH_FUSED
- Epochs: 1.0
- Mixed Precision: Native AMP
Performance Metrics (on evaluation set)
During training, the model achieved a validation loss of 1.2383. The training process involved 9000 steps across 1.0 epoch, with a total batch size of 64 across 8 devices.
Intended Use Cases
While specific intended uses and limitations are not detailed in the provided README, its fine-tuned nature suggests suitability for tasks similar to those present in the polyalign_dist_sft_train dataset. Developers should evaluate its performance for their specific applications, particularly where a compact Gemma 2-based model with specialized SFT is beneficial.