Overview

FelixChao/Voldemort-10B is a 10.7 billion parameter language model developed by FelixChao. It is a merged model, combining the architectures and weights of two distinct 7B models: FelixChao/WizardDolphin-7B and SanjiWatsuki/Silicon-Maid-7B. This merge was performed using a passthrough method, specifically layering different ranges from each source model to create a new, consolidated model.

Key Characteristics

• Parameter Count: 10.7 billion parameters, offering a balance between performance and computational requirements.
• Context Length: Supports a context window of 4096 tokens, enabling it to process and generate moderately long texts.
• Merge Method: Utilizes a passthrough merge, combining specific layer ranges from its base models (layers 0-24 from WizardDolphin-7B and layers 8-32 from Silicon-Maid-7B).
• Data Type: Configured to use bfloat16 for efficient computation.

Potential Use Cases

Given its merged nature and parameter count, Voldemort-10B is suitable for a range of general-purpose NLP tasks. Developers can leverage it for:

• Text generation and completion.
• Chatbot development and conversational AI.
• Summarization and information extraction.
• Fine-tuning for specific domain applications where the combined strengths of its base models are beneficial.