BayesRL/Llama3.1-IVON-SFT-8B
BayesRL/Llama3.1-IVON-SFT-8B is a supervised-fine-tuned language model based on the Qwen2.5-Math-7B foundation model. It was fine-tuned using the IVON variational optimizer, which yields an approximate Gaussian posterior over weights, serving as a learned prior for parameter-space exploration in RLVR (Reinforcement Learning with Variational Regularization) runs. This model is specifically designed as a warm-start checkpoint for advanced RLVR methods like 3PO, enabling exploration through weight perturbations sampled from its learned posterior.
Loading preview...
Model Overview
BayesRL/Llama3.1-IVON-SFT-8B is a supervised-fine-tuned (SFT) language model built upon the Qwen2.5-Math-7B foundation model. Its development is detailed in the paper "Parameter Exploration for RLVR via Variational Learning" (arXiv:2608.09805).
Key Characteristics
- Variational Optimization: This model was fine-tuned using the IVON (Variational Optimizer), which is distinct from standard optimizers. IVON produces not just point weights but an approximate Gaussian posterior, including a mean and a diagonal Hessian/precision estimate.
- Warm-Start Checkpoint: It functions as a "warm-start checkpoint." The learned Gaussian posterior serves as a prior to seed subsequent 3PO RLVR (Reinforcement Learning with Variational Regularization) runs. This allows for parameter-space exploration by sampling weight perturbations from this posterior.
- Training Data: The SFT process utilized the SFT subset of the Llama-Nemotron Post-Training Dataset.
Intended Use
This model is primarily intended as a foundational component for researchers and developers working with advanced RLVR techniques. It provides a specialized prior for systems like B3PO, M3PO, or C3PO, facilitating more effective parameter exploration in reinforcement learning contexts. It loads as a standard causal language model for inference, but its full potential is realized when integrated with the companion c3po code for RLVR experiments.