Model Overview

MarcoHermes is a 7 billion parameter language model developed by Eric111. It is a merged model, created using the mergekit tool, combining two distinct base models: AtAndDev/CapybaraMarcoroni-7B and eren23/DistilHermes-2.5-Mistral-7B.

Key Characteristics

• Architecture: A blend of two 7B parameter models, leveraging their respective strengths.
• Merge Method: Utilizes the slerp (spherical linear interpolation) merge method, with specific parameter weighting applied to self-attention and MLP layers to optimize performance.
• Context Length: Supports a context window of 4096 tokens.
• Data Type: Configured to use bfloat16 for efficient computation.

Use Cases

This model is suitable for general language generation tasks, benefiting from the combined capabilities of its merged components. Developers looking for a 7B model that integrates features from both CapybaraMarcoroni and DistilHermes-2.5-Mistral may find MarcoHermes a compelling option for various NLP applications.