Hercules-Mini-1.8B Overview

The Hercules-Mini-1.8B is a 1.8 billion parameter language model developed by M4-ai. It is a fine-tuned version of Qwen1.5-1.8B, leveraging 700,000 examples from Locutusque's Hercules-v4 dataset to enhance its capabilities.

Key Capabilities

• Multifaceted Performance: Demonstrates capabilities across various domains including math, coding, and function calling.
• Roleplay: Suitable for applications requiring role-playing scenarios.
• General Purpose: Designed for use as a general-purpose assistant, question answering, and chain-of-thought reasoning.
• Context Length: Supports a substantial context length of 32768 tokens.

Training Details

The model was fine-tuned using the Locutusque/hercules-v4.0 dataset. Training was conducted using bf16 non-mixed precision on 8 Kaggle TPUs, with a global batch size of 256 and a sequence length of 1536.

Limitations

Users should be aware that the end-of-sequence (EOS) token was not properly configured, requiring the implementation of a stopping criterion for the <|im_end|> token to prevent infinite generation.