Azure_Dusk-v0.2 Overview

Azure_Dusk-v0.2 is a 12 billion parameter language model developed by Epiculous, building upon the Mistral-Nemo-Base-2407 architecture. This version represents a significant advancement over its predecessor, Crimson_Dawn-v0.2, incorporating a substantially larger training dataset and leveraging RSLoRA for improved efficiency and performance during fine-tuning.

Key Enhancements & Training Details

• Base Model: Built on Mistral-Nemo-Base-2407.
• Training Methodology: Utilizes RSLoRA (as opposed to standard LoRA in previous versions) and incorporates a two-phased training approach over two epochs each on RP data and instruct data.
• Prompting Format: Trained exclusively on ChatML format, which is crucial for optimal interaction and performance.
• Hardware: Training was conducted on two NVIDIA A6000 GPUs.

Performance Metrics

Evaluations on the Open LLM Leaderboard show an average score of 14.03. Specific scores include:

• IFEval (0-Shot): 34.67
• BBH (3-Shot): 17.40
• MMLU-PRO (5-shot): 22.60

Usage and Prompting

Users should adhere strictly to the ChatML prompting structure for best results. Example:

<|im_start|>user
Hi there!<|im_end|>
<|im_start|>assistant
Nice to meet you!<|im_end|>

Quantized versions (exl2, gguf) are available for broader deployment.