conorfhayes/Qwen2.5-Math-7B-ES-MATH

Qwen2.5-Math-7B-ES-MATH Overview

This model is a 7.6 billion parameter Qwen/Qwen2.5-Math-7B variant, post-trained by conorfhayes using Evolution Strategies (ES). The primary goal of this training was to improve mathematical reasoning by enhancing solution coverage and pass@1, as detailed in the paper "Beyond the Best Guess: Improving LLM Solution Coverage with Evolution Strategies". Unlike traditional reinforcement learning, ES optimizes directly in weight space through random perturbations, aiming to broaden the model's output distribution while maintaining high accuracy.

Key Characteristics & Training

• Evolution Strategies (ES): Utilizes a population-based, gradient-free method for post-training, focusing on improving solution coverage rather than just the single best guess.
• Mathematical Reasoning Focus: Specifically trained on the MATH dataset (levels 3-5) with binary, verifier-checked correctness as the reward signal.
• RL-Zero Setting: Applied directly to the base model without prior supervised fine-tuning or distillation.
• Sampling-Oriented: Designed for repeated sampling (e.g., temperature 0.6, top-p 0.95) to fully leverage its broader output distribution; greedy decoding is not recommended.
• Prompt Format: Expects the Qwen2.5-Math prompt format with reasoning instructions in the system turn.

Limitations

• Domain Specific: Trained and evaluated exclusively on English mathematical reasoning; performance in other domains or languages is uncharacterized.
• Research Artefact: Intended for research into solution coverage and not designed as a deployment-ready assistant, lacking safety alignment.
• Outcome-Based Reward: Correct answers were rewarded even if the reasoning path was faulty.