Sao10K/L3-8B-Stheno-v3.2

Model Overview

Sao10K/L3-8B-Stheno-v3.2 is an 8 billion parameter model, fine-tuned by Sao10K from a Llama-3 base. This version, Stheno-v3.2-Zeta, represents the sixth iteration of development, focusing on refining its generative capabilities across various tasks. The training involved approximately 24 hours on a single H100 SXM GPU, with hyperparameter tuning to achieve lower loss levels.

Key Enhancements & Capabilities

• Balanced Content Generation: Improved handling of both SFW (Safe For Work) and NSFW (Not Safe For Work) content, achieving a more balanced output compared to previous versions.
• Enhanced Storywriting & Narration: Demonstrates better performance in generating coherent and engaging stories and narrative passages.
• Improved Assistant-Style Tasks: More effective at handling instruction-based and assistant-type queries.
• Multi-Turn Coherency: Features better consistency and reduced issues in multi-turn conversations, maintaining context more effectively.
• Prompt Adherence: Shows stronger adherence to given prompts and instructions, leading to more predictable and controlled outputs.

Training Data & Methodology

This version incorporated a mix of SFW and NSFW storywriting data, including contributions from Gryphe's Opus-WritingPrompts dataset. It also included more instruct/assistant-style data and underwent further cleaning of roleplaying samples to remove low-quality entries. While slightly less creative than its predecessor, this is considered a worthwhile trade-off for improved control and coherency.

Recommended Usage

Users are advised to use specific sampling parameters for optimal performance:

• Temperature: 1.12-1.22
• Min-P: 0.075
• Top-K: 50
• Repetition Penalty: 1.1

The model utilizes the Llama-3-Instruct prompting template and includes recommended stopping strings for effective interaction.