Overview

nbeerbower/gemma2-gutenberg-9B is a 9 billion parameter language model built upon the Gemma-2 architecture, specifically fine-tuned from the UCLA-AGI/Gemma-2-9B-It-SPPO-Iter3 base model. This iteration underwent further optimization using the ORPO (Optimized Reward-based Policy Optimization) method over three epochs, leveraging an RTX 4090 GPU. The fine-tuning process utilized the jondurbin/gutenberg-dpo-v0.1 dataset, which suggests a specialization in processing and generating text with characteristics found in literary works.

Key Capabilities & Performance

This model demonstrates capabilities across various general language understanding and reasoning tasks, as indicated by its performance on the Open LLM Leaderboard. Key evaluation metrics include:

• Avg. Score: 22.61
• IFEval (0-Shot): 27.96
• BBH (3-Shot): 42.36
• MMLU-PRO (5-shot): 35.47

While showing solid performance in areas like Big-Bench Hard and MMLU-PRO, its score on MATH Lvl 5 (1.44) suggests it is not primarily optimized for complex mathematical reasoning. The model's fine-tuning on a Gutenberg-derived dataset implies potential strengths in tasks requiring nuanced language understanding, stylistic generation, or analysis of literary content.

When to Consider This Model

This model is particularly suitable for use cases that benefit from a foundation in literary data, such as:

• Generating creative text or stories.
• Analyzing and summarizing long-form textual content.
• Applications requiring a model with a broad understanding of general language patterns and styles.
• Experiments with ORPO fine-tuning on a Gemma-2 base.