FelixChao/Voldemort-10B
FelixChao/Voldemort-10B is a 10.7 billion parameter language model created by FelixChao, formed by merging FelixChao/WizardDolphin-7B and SanjiWatsuki/Silicon-Maid-7B. This model leverages a passthrough merge method to combine the strengths of its constituent models, offering a versatile base for various natural language processing tasks. With a context length of 4096 tokens, it is suitable for applications requiring moderate input and output lengths.
Loading preview...
Overview
FelixChao/Voldemort-10B is a 10.7 billion parameter language model developed by FelixChao. It is a merged model, combining the architectures and weights of two distinct 7B models: FelixChao/WizardDolphin-7B and SanjiWatsuki/Silicon-Maid-7B. This merge was performed using a passthrough method, specifically layering different ranges from each source model to create a new, consolidated model.
Key Characteristics
- Parameter Count: 10.7 billion parameters, offering a balance between performance and computational requirements.
- Context Length: Supports a context window of 4096 tokens, enabling it to process and generate moderately long texts.
- Merge Method: Utilizes a
passthroughmerge, combining specific layer ranges from its base models (layers 0-24 from WizardDolphin-7B and layers 8-32 from Silicon-Maid-7B). - Data Type: Configured to use
bfloat16for efficient computation.
Potential Use Cases
Given its merged nature and parameter count, Voldemort-10B is suitable for a range of general-purpose NLP tasks. Developers can leverage it for:
- Text generation and completion.
- Chatbot development and conversational AI.
- Summarization and information extraction.
- Fine-tuning for specific domain applications where the combined strengths of its base models are beneficial.