kushalpatil/jevify-gemma4-e4b
kushalpatil/jevify-gemma4-e4b is a 7.9 billion parameter Gemma-4-E4B-it model, fine-tuned by kushalpatil using LoRA, specifically designed to provide honest probabilities for typed questions about a given state. This model excels at probabilistic decision-making, serving as the core of the Jevify API. It is optimized for tasks requiring accurate probability distributions over answer labels rather than text generation, making it suitable for structured data analysis and classification.
Loading preview...
Model Overview
kushalpatil/jevify-gemma4-e4b is a fine-tuned version of google/gemma-4-E4B-it, specifically optimized for probabilistic decision-making. This 7.9 billion parameter model is designed to output honest probabilities when presented with typed questions about a piece of state, rather than generating free-form text. It is the underlying model for the jevify project, which provides a local, Jev-compatible probabilistic decision API.
Key Capabilities
- Probabilistic Output: Provides next-token distributions over predefined answer labels, ideal for classification and structured data analysis.
- Jev-compatible: Integrates seamlessly with the Jevify framework for structured queries and responses.
- Efficient Inference: Designed for single prefill operations to obtain probability distributions.
Training Details
The model was fine-tuned using LoRA (r=64 on attention projections) over two epochs on approximately 47,000 items. The training dataset included diverse sources such as hard-labeled classification sets, multi-annotator sets with human label distributions, and constructed long states (up to 24k tokens). The training objective minimized KL divergence between target and label distributions, without relying on a teacher model.
Performance Highlights
Evaluations on held-out, out-of-distribution datasets (6 datasets, 307 items) show strong performance, with an overall accuracy of 84.4% and a low ECE of 0.043. In-distribution held-out data (800 items) demonstrated significant improvement from raw to trained states, with accuracy increasing from 74.5% to 82.3% and ECE dropping from 0.217 to 0.028.
Good For
- Applications requiring calibrated probability scores for classification tasks.
- Integrating with systems that need structured, probabilistic answers to questions about a given state.
- Use cases where interpretability of probabilities is crucial, such as decision support systems.