Dans-Archive/Dans-PileOfSets-Mk1-llama-13b-merged
Dans-PileOfSets-Mk1-llama-13b-merged is a 13 billion parameter Llama-based model developed by Dans-Archive, created by merging a LoRA. This model is specifically trained to maintain a logical thread in its output, making it particularly effective for storytelling and writing tasks. It was trained on a reduced version of the Pile of Sets dataset, focusing on coherence and narrative consistency.
Loading preview...
Model Overview
Dans-PileOfSets-Mk1-llama-13b-merged is a 13 billion parameter Llama-based model, developed by Dans-Archive, that integrates a LoRA merge. The primary objective behind its creation is to produce text that maintains a strong logical thread, whether the output is verbose or concise. This model is considered a prototype, with future iterations planned to enhance its capabilities.
Key Capabilities
- Logical Coherence: Designed to uphold a consistent logical flow in generated text.
- Storytelling and Writing Aid: Optimized for creative writing and narrative generation, serving as a tool for authors.
- Alpaca Prompt Format: Utilizes the Stanford Alpaca prompt format for both instruction-only and instruction-with-input scenarios.
Training Details
The model underwent 2 epochs of training, utilizing a reduced version of the Pile of Sets dataset (40% of its original size) to accelerate training iterations. Training involved various cleaned and culled datasets, including Camel biology, physics, chemistry, math, and AI society, Alpaca evol instruct, GPTeacher Instruct, Alpaca GPT4, and Dolly Databricks. Perplexity benchmarks on wikitext show a score of 4.66796875.
Future Plans
Dans-Archive plans to pivot to a conversational format, train a Mk2 version against the entirety of the Pile of Sets, and develop a 30B parameter Mk3 model with expanded story generation capabilities.