TheBloke/Selfee-13B-SuperHOT-8K-fp16
TheBloke/Selfee-13B-SuperHOT-8K-fp16 is a 13 billion parameter language model, a merge of Kaist AI's Selfee 13B and Kaio Ken's SuperHOT 8K LoRA. This model is designed for extended context applications, supporting an 8K context length through a specialized scaling technique. It is particularly optimized for scenarios requiring longer conversational memory or processing of extensive documents, building on Selfee's iterative self-revising capabilities and SuperHOT's context extension.
Loading preview...
Model Overview
This model, TheBloke/Selfee-13B-SuperHOT-8K-fp16, is a 13 billion parameter language model created by merging Kaist AI's Selfee 13B with Kaio Ken's SuperHOT 8K LoRA. The primary differentiator is its 8K context length, achieved through a specific scaling technique that can be activated using trust_remote_code=True during inference. The config.json is pre-set for an 8192 sequence length, though it can be adjusted.
Key Capabilities
- Extended Context Window: Supports an 8K (8192 token) context length, enabling longer interactions and processing of more extensive texts.
- Iterative Self-Revision: Inherits Selfee's unique training methodology, which involves generating iterative self-feedback and revisions until a satisfactory response is achieved. This can operate in autonomous or enforced revision modes.
- Merged Architecture: Combines the base capabilities of Kaist AI's Selfee 13B, an instruction-following LLaMA model, with the context extension of Kaio Ken's SuperHOT 8K.
Good For
- Applications requiring long-form content generation or analysis.
- Use cases where iterative refinement of responses is beneficial.
- Scenarios demanding a larger memory for conversational AI or document processing.
Limitations
Similar to other LLaMA-finetuned models, it may exhibit limitations in complex math, reasoning, factual accuracy, and coding tasks. The evaluation methodology for Selfee also notes limitations in comprehension and potential inconsistencies.