icefog72/IceTeaRP-7b

IceTeaRP-7b: A 7B Parameter Roleplay-Optimized Model

IceTeaRP-7b is a 7 billion parameter language model developed by icefog72, created through a SLERP merge of icefog72/Kunokukulemonchini-7b and a BigLM7-7b merge (which itself combines liminerity/M7-7b and Undi95/BigL-7B). This model is specifically designed for roleplay applications, aiming to provide a more coherent and capable experience than its predecessors.

Key Capabilities & Features

• Extended Context Window: Capable of handling a 32k context window without requiring additional scaling, making it suitable for long-form interactions.
• Alpaca Prompt Template: Utilizes the Alpaca prompt template for instruction following.
• Merge Method: Developed using the SLERP (Spherical Linear Interpolation) merge method, which combines the strengths of its constituent models.
• Quantized Versions Available: Provided in various EXL2 quantized versions (4.0bpw, 4.2bpw, 6.5bpw, 8.0bpw) for efficient deployment.

Performance & Considerations

While designed for extended contexts, user feedback indicates that the model may develop repetition issues at 16k-32k context lengths without well-structured roleplay rules or Chain-of-Thought (CoT) prompting. Adjusting the rope_theta parameter (e.g., to 60000.0) is suggested as a potential method to enhance coherence.

Open LLM Leaderboard Evaluation

IceTeaRP-7b demonstrates competitive performance across various benchmarks:

• Avg.: 69.76
• AI2 Reasoning Challenge (25-Shot): 66.98
• HellaSwag (10-Shot): 86.13
• MMLU (5-Shot): 63.97
• TruthfulQA (0-shot): 62.44
• Winogrande (5-shot): 78.85
• GSM8k (5-shot): 60.20

Use Cases

This model is particularly well-suited for:

• Roleplay Scenarios: Its design and context handling make it ideal for interactive storytelling and character-driven applications.
• Applications Requiring Long Context: Useful for tasks that benefit from maintaining extensive conversational history or processing large documents.