Overview
BeaverAI/MS-2501-DPE-QwQify-v0.1-24B is a 24 billion parameter language model developed by BeaverAI, built upon PocketDoc/Dans-PersonalityEngine-V1.2.0-24b, which originates from Mistral-Small-24B-Base-2501. The primary goal of this fine-tuning is to integrate a distinct 'QwQ' personality into the base model, making it suitable for interactive and character-driven applications. It supports a context length of 32768 tokens.
Key Capabilities
- Personality Integration: Designed to imbue an existing general-purpose instruct model with specific personality traits, referred to as 'QwQ's thoughts'.
- ChatML Formatting: Utilizes ChatML for prompt formatting, requiring users to manage previous turns by removing 'thinking' elements.
- Conversational Nuance: May require initial regeneration or manual editing of responses in the first few turns to align the model's output with desired conversational style.
Training Details
The model was fine-tuned using LoRA (Low-Rank Adaptation) with a learning rate of 2e-05 over 2 epochs. The training involved a diverse set of datasets, including those focused on character cards, chat logs, and augmented system prompts, all processed with a custom ChatML regex strategy. The training process achieved a final validation loss of 1.1949.
Good For
- Character-based AI: Ideal for applications where the AI needs to adopt a specific, consistent personality or persona.
- Interactive Storytelling: Can be used in scenarios requiring dynamic and personality-driven dialogue generation.
- Custom Conversational Agents: Suitable for developers looking to create chatbots with unique and predefined interaction styles.