BayesRL/Qwen2.5Math-IVON-SFT-7B
BayesRL/Qwen2.5Math-IVON-SFT-7B is a 7 billion parameter language model based on the Qwen2.5-Math architecture, developed by BayesRL. It is supervised-fine-tuned using the IVON variational optimizer, which yields a Gaussian posterior over weights rather than point weights. This model serves as a warm-start checkpoint, providing a learned prior for parameter-space exploration in subsequent 3PO RLVR runs. Its primary use is to seed advanced reinforcement learning from human feedback (RLHF) experiments, particularly for mathematical reasoning tasks.
Loading preview...
Model Overview
BayesRL/Qwen2.5Math-IVON-SFT-7B is a specialized 7 billion parameter language model derived from the Qwen/Qwen2.5-Math-7B foundation model. It has undergone Supervised Fine-Tuning (SFT) using the IVON (Variational Optimizer), a method detailed in the paper "Parameter Exploration for RLVR via Variational Learning" (arXiv:2608.09805).
Key Characteristics
- Variational Optimization: Unlike standard SFT that produces fixed weights, IVON generates an approximate Gaussian posterior (mean and diagonal Hessian/precision estimate) over the model's parameters.
- Warm-Start Checkpoint: This model functions as a crucial warm-start prior. The learned posterior is used to seed subsequent 3PO RLVR (Reinforcement Learning from Variational Rewards) runs, enabling parameter-space exploration by sampling weight perturbations.
- Foundation for RLVR: It is designed to be the initial state for advanced RLVR experiments, specifically for models like Qwen-B3PO-7B, Qwen-M3PO-7B, and Qwen-C3PO-7B.
Training Details
The model was trained on 8 NVIDIA H200 GPUs (144 GB each) using the IVON optimizer. The associated code for utilizing this model as a warm-start prior for 3PO RLVR is available in the insait-institute/c3po repository.
Intended Use
This model is primarily intended for researchers and developers working on advanced RLHF techniques, particularly those exploring variational learning and parameter-space exploration for improving model performance in mathematical and reasoning tasks.