Sao10K/70B-L3.3-mhnnn-x1
Sao10K/70B-L3.3-mhnnn-x1 is a 70 billion parameter language model with a 32,768 token context length, developed by Sao10K. This model is fine-tuned with a unique data composition, similar to 'Freya', focusing on creative outputs across various tasks. It excels in completion tasks, text adventures, amoral assistant roles, and general instruction following, offering more creative responses at the cost of occasional inconsistencies.
Loading preview...
Model Overview
Sao10K/70B-L3.3-mhnnn-x1 is a 70 billion parameter language model with a 32,768 token context length, developed by Sao10K. It was trained for approximately 14 hours on an 8xH100 node. The model utilizes a Llama-3-Instruct prompt format and is noted for its creative output, though this can sometimes lead to 'brainfarts' that are typically resolved with regeneration.
Key Capabilities
- Creative Completion: Optimized for generating novels and eBooks.
- Text Adventure Narrator: Capable of acting as a text adventure narrator when prompted with specific system instructions and a one-shot example.
- Amoral Assistant: Can function as an amoral or neutral assistant by including specific terms in the system prompt.
- General Instruction Following: Handles standard assistant tasks and roleplay scenarios.
Training Details
The model's training data composition is similar to 'Freya' but applied differently. It includes a mix of completion data (eBooks, novels) and chat template data (amoral assistant, Hespera, RPG adventure instructions). The training utilized axolotl with LoRA adaptation, featuring lora_r: 64, lora_alpha: 64, and lora_dropout: 0.2. It also incorporates Liger plugins for enhanced rope, RMS norm, layer norm, and GLU activation.
Top 3 parameter combinations used by Featherless users for this model. Click a tab to see each config.