Hugging Face Overview
ArliAI/Llama-3.1-8B-ArliAI-RPMax-v1.3 Overview
ArliAI's Llama-3.1-8B-ArliAI-RPMax-v1.3 is an 8 billion parameter model, part of the RPMax series, specifically fine-tuned from the Llama-3.1-8B-Instruct base. This model is engineered to enhance creativity and reduce repetition in generated text, particularly for creative writing and roleplay scenarios. It utilizes a context length of 128K tokens.
Key Differentiators & Capabilities
- Reduced Cross-Context Repetition: The primary goal of RPMax is to eliminate the model's tendency to repeat phrases or tropes across different conversations, ensuring more unique and varied outputs.
- Highly Creative Outputs: Achieved through a unique dataset curation process that emphasizes variety and deduplication, preventing the model from overfitting to specific personalities or situations.
- Unconventional Training Approach: Trained for a single epoch with a low gradient accumulation and a higher learning rate, allowing the model to learn from individual examples without reinforcing repetitive patterns.
- Updated Training Enhancements: Version 1.3 incorporates fixes for the gradient checkpointing bug and utilizes RSLORA+ for improved learning and output quality.
Training Details
- Dataset Curation: RPMax models are trained on a diverse set of curated creative writing and roleplay datasets, meticulously deduplicated to ensure no two entries share repeated characters or situations. Synthetic generations are excluded to avoid "GPT-isms."
- Training Parameters: Features a sequence length of 8192, 1 epoch training, RS-QLORA+ (64-rank 64-alpha) with ~2% trainable weights, and a learning rate of 0.00001.
Good for
- Creative Writing: Generating diverse and imaginative narratives.
- Roleplay (RP): Engaging in dynamic and unpredictable character interactions.
- Interactive Storytelling: Applications requiring varied and non-repetitive responses from an AI character.