KexuanShi/sft_gemma2_2b_layerdrop_attn_p005
KexuanShi/sft_gemma2_2b_layerdrop_attn_p005 is a 2.6 billion parameter language model fine-tuned using the TRL framework. This model is based on the Gemma 2 architecture and incorporates LayerDrop with an attention probability of 0.05, suggesting an optimization for efficiency or specific performance characteristics. It is designed for general text generation tasks, particularly instruction-following, as indicated by its SFT (Supervised Fine-Tuning) training method.
Loading preview...
Model Overview
KexuanShi/sft_gemma2_2b_layerdrop_attn_p005 is a 2.6 billion parameter language model that has undergone supervised fine-tuning (SFT) using the TRL (Transformers Reinforcement Learning) library. This model is built upon the Gemma 2 architecture and integrates a LayerDrop mechanism with an attention probability of 0.05, which typically aims to improve model efficiency or generalization by randomly dropping layers during training.
Key Characteristics
- Architecture: Based on the Gemma 2 model family.
- Parameter Count: 2.6 billion parameters.
- Training Method: Supervised Fine-Tuning (SFT) for instruction-following capabilities.
- Optimization: Utilizes LayerDrop with an attention probability of 0.05, suggesting a focus on robust or efficient performance.
- Context Length: Supports a context window of 8192 tokens.
Intended Use Cases
This model is suitable for various text generation tasks where instruction-following is beneficial. Developers can leverage it for:
- General Text Generation: Creating coherent and contextually relevant text based on prompts.
- Instruction Following: Responding to user queries or instructions in a structured manner.
- Exploration of Efficient Architectures: As it incorporates LayerDrop, it could be of interest for research into more efficient large language models.