Hugging Face Overview
Overview
crestf411's L3.1-8B-Slush-v1.1 is an 8 billion parameter model built upon the Llama 3.1 architecture, designed to address the base model's limitations in creativity and imagination. It employs a unique two-stage training methodology:
- Stage 1 (Pretraining Continuation): Focuses on boosting the model's creativity and writing abilities through high LoRA dropout, then merging this into the instruction-tuned base.
- Stage 2 (Fine-tuning): Further refines roleplaying capabilities and mitigates any potential degradation from the initial merge.
This model incorporates a custom merge using MergeKit with the TIES method, targeting the meta-llama/Llama-3.1-8B as its base. Training parameters were adjusted in v1.1 based on feedback, including specific LoRA configurations (rank 64, alpha 128 for stage 1; rank 32, alpha 64 for stage 2) and the use of LoRA+.
Key Capabilities
- Enhanced Creativity: Aims to improve imaginative text generation.
- Improved Writing: Designed for better overall writing quality.
- Strong Roleplaying: Specifically fine-tuned to excel in interactive roleplay scenarios, following the Silly Tavern preset.
Good For
- Applications requiring creative text generation.
- Interactive storytelling and roleplaying.
- Use cases where imagination and nuanced responses are critical.