Model Overview

BeaverAI's mistral-doryV2-12b is a 12 billion parameter instruction-tuned language model, built upon the Mistral Nemo 12B base. This version, Dory v2, represents a re-finetune focused on general instruction following rather than specific conversational or role-play applications. It supports a substantial context length of 32768 tokens, allowing for processing and generating longer texts.

Training Details

The model was trained using Rank 64 QDoRA and leveraged a curated mix of datasets to enhance its instruction-following capabilities. Key datasets included:

• kalomaze/Opus_Instruct_3k: A comprehensive instruction dataset.
• Magpie-Align/Magpie-Gemma2-Pro-Preview-Filtered: Conversations with high reward model ratings.
• Gryphe/Sonnet3.5-SlimOrcaDedupCleaned: A cleaned and deduplicated dataset.
• Fizzarolli/FallingThroughTheSkies-592k-Filtered-Filtered: High-rated stories published before 2020.

Prompting Format

The model utilizes an Alpaca-like prompting structure, which includes optional system prompts, clear instruction sections for queries, and designated response areas. This format ensures straightforward interaction and consistent output generation.

Key Differentiator

Unlike some other models in its class, mistral-doryV2-12b is explicitly not focused on (E)RP-style interactions, aiming instead for broader utility in general instruction-based tasks.