bosonai/Higgs-Llama-3-70B
Higgs-Llama-3-70B is a 70 billion parameter language model developed by bosonai, post-trained from Meta-Llama-3-70B with an 8192-token context length. It is specifically tuned for role-playing scenarios while maintaining strong general-domain instruction-following and reasoning capabilities. The model utilizes supervised fine-tuning and iterative preference optimization, including a special strategy for aligning behavior with system messages. It demonstrates competitive performance on challenging benchmarks like MMLU-Pro and Arena-Hard, often outperforming its base model.
Loading preview...
Higgs-Llama-3-70B: Role-Playing and Instruction-Following
Higgs-Llama-3-70B, developed by bosonai, is a 70 billion parameter model derived from Meta-Llama-3-70B, featuring an 8192-token context window. Its primary distinction lies in its specialized tuning for role-playing, achieved through supervised fine-tuning with in-house datasets and iterative preference optimization. This process includes a unique strategy to ensure the model adheres closely to system messages, making it particularly effective for character-driven interactions.
Key Capabilities & Performance
- Enhanced Role-Playing: Designed to follow roles more consistently and accurately compared to other instruct models.
- Strong General Instruction-Following: Maintains competitive performance in general instruction-following and reasoning tasks.
- Benchmark Performance: Achieves a 63.2 score on MMLU-Pro and 49.6 on Arena-Hard, outperforming the base Llama-3-70B-Instruct model on these challenging, recently released benchmarks. It also shows improvements on AlpacaEval 2.0 LC, MMLU, GPQA, and DROP.
Training Methodology
The model underwent supervised fine-tuning using proprietary instruction-following and chat datasets. A semi-automated pipeline, combining human labelers and private LLMs, was used to construct preference pairs for iterative preference optimization. The training specifically aimed to exclude benchmark data to prevent overfitting and improve role-playing performance.
Usage
Higgs-Llama-3-70B uses the same prompting format as Meta-Llama-3-70B-Instruct, making it straightforward to integrate into existing Llama 3 workflows. Example code for use with the transformers library is provided, demonstrating how to apply chat templates and generate responses with specific system messages for role-playing.