Overview

Mistral-7B-ReMax-v0.1 is a 7 billion parameter language model developed by Ziniu Li, building upon the Mistral-7B-Instruct-v0.2 architecture. Its key differentiator is the application of the ReMax algorithm for Reinforcement Learning from Human Preference (RLHF) fine-tuning, detailed in a research paper. This fine-tuning process aims to enhance the model's ability to generate high-quality, instruction-following responses.

Key Capabilities & Performance

• Improved Response Quality: Fine-tuned using 10k prompts from the lmsys-chat-1m dataset, Mistral-7B-ReMax-v0.1 shows a 57.8% win-rate against its base model (Mistral-7B-Instruct-v0.2) in GPT-4 evaluated comparisons.
• Enhanced Instruction Following: The model maintains the same instruction format as Mistral-7B-Instruct-v0.2, using [INST] and [/INST] tokens for prompts.
• Benchmark Performance: On AlpacaEval, Mistral-7B-ReMax-v0.1 achieves a score of 94.40 (AlpacaEval 2: 16.00), outperforming Mistral-7B-Instruct-v0.2 and LLaMa2 Chat 7B, and approaching GPT-4's performance in some metrics.

Training Details

The model was fine-tuned using the ReMax algorithm, leveraging the UltraRM-13b as its reward model. The training procedure involved adjusting the chat template for reward score evaluation to align with UltraRM-13b's format.

Limitations

Mistral-7B-ReMax-v0.1 shares the same limitations as its base model, Mistral-7B-Instruct-v0.2, notably lacking built-in moderation mechanisms.