Gemmalpaca-2B: An Enhanced Gemma-2B Variant
mlabonne/Gemmalpaca-2B is a 2.6 billion parameter model derived from the Gemma-2B base, specifically fine-tuned using the vicgalle/alpaca-gpt4 dataset. This supervised fine-tuning process has resulted in a model that outperforms Google's official gemma-2b-it (instruction-tuned) version on several key benchmarks.
Key Capabilities & Performance
- Superior Benchmark Scores: Achieves an average score of 38.39 on Nous' benchmark suite, surpassing
gemma-2b-it (36.1) and gemma-2b (34.26). Notable improvements are seen in AGIEval and GPT4All metrics. - Extended Context Window: Features an 8192-token context length, allowing for more extensive conversational turns and complex prompts.
- Instruction Following: Designed for instruction-following tasks, leveraging the Alpaca-GPT4 dataset for robust response generation.
Recommended Usage
- Alpaca Chat Template: It is recommended to use this model with the Alpaca chat template rather than the Gemma Instruct template for optimal performance.
- Stop Token: Ensure
</s> is added as a stop token for proper generation termination. - Quantized Versions: GGUF quantized models are available for efficient local deployment.
This model serves as a strong demonstration of effective fine-tuning on Gemma architecture, offering enhanced performance for general-purpose conversational AI in its size class.