minjujeon/mistral-7b-arc-dpo

Model Overview

minjujeon/mistral-7b-arc-dpo is a 7 billion parameter Mistral-7B-v0.1 model specifically fine-tuned for the ARC-Challenge benchmark, a deep-learning coding test from Upstage AIRE. It demonstrates a notable improvement over the base model, achieving an acc_norm of 71.84 ± 0.48 compared to the base model's 61.07 ± 0.36 under the official 25-shot protocol.

Training Recipe

The model was trained in two stages using LoRA (r=32, alpha=64, all seven projections):

1. Supervised Fine-Tuning (SFT): Performed on ARC-Challenge + ARC-Easy train splits (3,370 items), with loss masked to the answer span. This stage used a learning rate of 1e-4 cosine over 2 epochs.
2. Direct Preference Optimization (DPO): Applied to 8,025 gold-vs-distractor pairs. These pairs were drawn from a synthetic pool of 2,675 teacher-generated (Qwen3.8-27B) MCQs that the SFT model initially failed, focusing on "hard mining" examples. DPO used a beta of 0.1 with hard-pair gating and a chosen-NLL anchor.

Performance Highlights

• Mistral-7B-v0.1 (base): 61.07 ± 0.36 acc_norm
• + SFT only: 67.04 ± 0.50 acc_norm
• + SFT + DPO (this model): 71.84 ± 0.48 acc_norm
• Qwen3.8-27B (teacher): 73.21 (single seed)

Use Cases

This model is particularly well-suited for:

• Scientific Question Answering: Excelling in tasks requiring complex reasoning and analytical problem-solving, as demonstrated by its performance on the ARC-Challenge benchmark.
• Educational AI Tools: Developing AI assistants that can help students with challenging science and reasoning questions.
• Knowledge Retrieval and Synthesis: Applications where accurate and nuanced understanding of factual information is critical.