Hugging Face Overview
Model Overview
Envoid/Llama-3.05-NT-Storybreaker-Ministral-70B is an experimental 70 billion parameter model created by Envoid, designed to transfer the unique roleplaying characteristics of the smaller Ministral-8B-Instruct-2410 onto a more capable Llama-3.05 base. This process, termed "reverse-distillation," involved fine-tuning the Llama-3.05-Nemotron-Tenyxchat-Storybreaker-70B model on a custom single-turn roleplaying dataset derived from the 'leaked undislop' dataset, processed through Ministral to capture its conversational style.
Key Capabilities & Training
- Enhanced Roleplaying: The model aims to improve narrative flow, characterization, and scenario adherence, moving beyond short conversational loops to a continuous flow of actions.
- Custom Dataset: Trained on a unique dataset formatted from existing roleplaying conversations, then run through Ministral to capture its specific interaction patterns.
- High Dropout Training: Utilizes an atypically high dropout rate (0.6) during QLoRA training, inspired by research on improving out-of-distribution performance.
- SLERP Merging: The fine-tuned LoRA adapter was merged back onto the original model using a 40/60 SLERP blend to integrate new behaviors while retaining original capabilities.
Considerations for Use
- Mature Content: This model is explicitly noted to be more inclined towards adult content and is geared for both SFW and NSFW roleplaying, intended for mature audiences.
- Temperature Sensitivity: Can be "sloppy" at higher temperatures and rigid at lower temperatures; lowering temperature is suggested to improve overall coherence.
- Experimental Nature: As an early attempt at reverse-distillation, results are based on subjective judgment, with potential for further improvement with more VRAM, higher LoRA rank, and increased sequence length during training.