Ateron/Gemma-4-MoonGem-31B

Overview

Ateron/Gemma-4-MoonGem-31B is a 31 billion parameter language model built upon the Gemma-4 architecture. This model was developed by Ateron through a sophisticated multi-phase merging process using Mergekit, integrating various base models to enhance its performance and capabilities. It supports a context length of 32768 tokens, making it suitable for processing longer inputs and generating more extensive outputs.

Mergekit Configuration

The model's creation involved a three-phase merging strategy:

• Phase 1 (Bleed): Utilized the ties merge method, combining Gemma-4-Glimmer, Gemma-4-Gutenberg, and Gemma-4-Gemopus with Gemma-4-Scotoma-V2 as the base model. This phase involved specific density and layer-wise weighting parameters.
• Phase 2 (Crimson): Employed the task_arithmetic merge method, integrating Gemma-4-Melinoe-VL, Gemma-4-MeroMero-V2, and Gemma-4-MusicaV1 with Gemma-4-Scotoma-V2 as the base.
• Phase 3 (Blood): Also used the task_arithmetic merge method, combining Gemma-4-MicroMix-P1 and Gemma-4-MicroMix-P2 with Gemma-4-Scotoma-V2 as the base.

Key Characteristics

• Architecture: Based on the Gemma-4 family.
• Parameter Count: 31 billion parameters.
• Context Length: 32768 tokens.
• Development Method: Advanced multi-phase merging using Mergekit, indicating a focus on combining strengths from multiple specialized models.

Intended Use Cases

Given its large parameter count and substantial context window, MoonGem-31B is suitable for a broad range of natural language processing tasks, including but not limited to:

• Complex text generation
• Detailed content creation
• Advanced conversational AI
• Summarization of long documents
• Question answering over extensive texts