IceTeaRP-7b: A 7B Parameter Roleplay-Optimized Model
IceTeaRP-7b is a 7 billion parameter language model developed by icefog72, created through a SLERP merge of icefog72/Kunokukulemonchini-7b and a BigLM7-7b merge (which itself combines liminerity/M7-7b and Undi95/BigL-7B). This model is specifically designed for roleplay applications, aiming to provide a more coherent and capable experience than its predecessors.
Key Capabilities & Features
- Extended Context Window: Capable of handling a 32k context window without requiring additional scaling, making it suitable for long-form interactions.
- Alpaca Prompt Template: Utilizes the Alpaca prompt template for instruction following.
- Merge Method: Developed using the SLERP (Spherical Linear Interpolation) merge method, which combines the strengths of its constituent models.
- Quantized Versions Available: Provided in various EXL2 quantized versions (4.0bpw, 4.2bpw, 6.5bpw, 8.0bpw) for efficient deployment.
Performance & Considerations
While designed for extended contexts, user feedback indicates that the model may develop repetition issues at 16k-32k context lengths without well-structured roleplay rules or Chain-of-Thought (CoT) prompting. Adjusting the rope_theta parameter (e.g., to 60000.0) is suggested as a potential method to enhance coherence.
Open LLM Leaderboard Evaluation
IceTeaRP-7b demonstrates competitive performance across various benchmarks:
- Avg.: 69.76
- AI2 Reasoning Challenge (25-Shot): 66.98
- HellaSwag (10-Shot): 86.13
- MMLU (5-Shot): 63.97
- TruthfulQA (0-shot): 62.44
- Winogrande (5-shot): 78.85
- GSM8k (5-shot): 60.20
Use Cases
This model is particularly well-suited for:
- Roleplay Scenarios: Its design and context handling make it ideal for interactive storytelling and character-driven applications.
- Applications Requiring Long Context: Useful for tasks that benefit from maintaining extensive conversational history or processing large documents.