ProtenLabs/gemma-4-E4B-it-image2text-ko-dpo

Model Overview

ProtenLabs/gemma-4-E4B-it-image2text-ko-dpo is a specialized vision-language model (VLM) designed to interpret chart images and provide detailed explanations in Korean. Built upon Google's gemma-4-E4B-it (8.08B multimodal base model), it underwent a two-stage fine-tuning process: initial QLoRA SFT, followed by Direct Preference Optimization (DPO) targeting specific failure cases.

Key Capabilities & Differentiators

• Chart Image-to-Text in Korean: Its primary function is to convert visual chart data into comprehensive Korean textual descriptions.
• Failure-Targeted DPO: A unique aspect is its DPO phase, which specifically used 870 "failure-targeted" preference pairs. Here, the 'chosen' response was the ground-truth explanation, and the 'rejected' response was the SFT model's greedy output that omitted some numerical values (recall < 0.9). This method aims to directly correct the model's common errors.
• Improved Recall on Failure Cases: This targeted DPO led to a significant improvement in recall for the identified failure cases, increasing from 60.9% to 65.8% (+4.9pp).
• Merged Full Model: The final model is a 16-bit merged full model, allowing for standalone loading without the base model.

Performance Highlights

While overall metrics (like LLM-as-judge scores) remain comparable to the SFT version due to the SFT's already high performance, the DPO process successfully shifted the greedy output distribution, with a 39.6% output change rate compared to SFT. This indicates a more robust and accurate output for previously problematic cases.

Use Cases

This model is ideal for applications requiring highly accurate and detailed Korean descriptions of chart images, particularly where numerical precision and comprehensive recall are critical. It can be integrated using vLLM for efficient inference.