BayesRL/Llama3.1-IVON-SFT-8B
BayesRL/Llama3.1-IVON-SFT-8B is an 8 billion parameter supervised fine-tuned language model based on the Qwen2.5-Math-7B architecture. It was fine-tuned using the IVON variational optimizer, which generates an approximate Gaussian posterior over model weights, serving as a learned prior for parameter-space exploration in RLVR (Reinforcement Learning from Variational Rewards) methods. This model functions as a warm-start checkpoint, providing a robust initial state for advanced RLVR training runs.
Loading preview...
Model Overview
BayesRL/Llama3.1-IVON-SFT-8B is an 8 billion parameter language model derived from Qwen/Qwen2.5-Math-7B. It has undergone a warm-start supervised fine-tuning (SFT) process utilizing the IVON (Variational Optimizer), as detailed in the paper "Parameter Exploration for RLVR via Variational Learning." This SFT stage is unique because IVON yields not just point weights, but an approximate Gaussian posterior distribution over these weights, comprising a mean and a diagonal Hessian/precision estimate.
Key Characteristics
- Foundation Model: Built upon
Qwen/Qwen2.5-Math-7B. - Optimization Method: Fine-tuned with the IVON optimizer, which is designed to explore parameter space effectively.
- Output: Produces a learned prior (Gaussian posterior) over weights, crucial for seeding subsequent 3PO RLVR (B3PO / M3PO / C3PO) runs.
- Purpose: Serves as a warm-start checkpoint for advanced Reinforcement Learning from Variational Rewards (RLVR) experiments, where weight perturbations sampled from the learned posterior drive exploration.
Usage and Integration
This model loads as a standard causal language model using Hugging Face Transformers. Its primary intended use is to provide an initial, well-calibrated prior for the 3PO RLVR framework, enabling more effective parameter-space exploration during RLVR training. The IVON optimizer state can be loaded for this purpose via specific environment variables in the companion c3po codebase.