BeaverAI/mistral-doryV2-12b
BeaverAI/mistral-doryV2-12b is a 12 billion parameter instruction-tuned causal language model, re-finetuned from Mistral Nemo 12B's base. It features a 32768 token context length and is optimized for general instruction following, distinguishing itself from models focused on specific conversational styles. The model was trained using QDoRA on a diverse dataset including instruction, reward-rated, and story-based conversations, making it suitable for a broad range of text generation tasks.
Loading preview...
Model Overview
BeaverAI's mistral-doryV2-12b is a 12 billion parameter instruction-tuned language model, built upon the Mistral Nemo 12B base. This version, Dory v2, represents a re-finetune focused on general instruction following rather than specific conversational or role-play applications. It supports a substantial context length of 32768 tokens, allowing for processing and generating longer texts.
Training Details
The model was trained using Rank 64 QDoRA and leveraged a curated mix of datasets to enhance its instruction-following capabilities. Key datasets included:
- kalomaze/Opus_Instruct_3k: A comprehensive instruction dataset.
- Magpie-Align/Magpie-Gemma2-Pro-Preview-Filtered: Conversations with high reward model ratings.
- Gryphe/Sonnet3.5-SlimOrcaDedupCleaned: A cleaned and deduplicated dataset.
- Fizzarolli/FallingThroughTheSkies-592k-Filtered-Filtered: High-rated stories published before 2020.
Prompting Format
The model utilizes an Alpaca-like prompting structure, which includes optional system prompts, clear instruction sections for queries, and designated response areas. This format ensures straightforward interaction and consistent output generation.
Key Differentiator
Unlike some other models in its class, mistral-doryV2-12b is explicitly not focused on (E)RP-style interactions, aiming instead for broader utility in general instruction-based tasks.
Top 3 parameter combinations used by Featherless users for this model. Click a tab to see each config.