TengfeiLiuCoder/RefCaptioner
RefCaptioner is an 8 billion parameter vision-language model developed by Tengfei Liu and collaborators, based on the Qwen3-VL-8B-Instruct architecture. It specializes in multi-reference image-grounded video captioning, generating fluent English descriptions of videos while precisely tagging visual phrases with corresponding reference images. This model is designed to ground relevant reference images to local phrases in a video caption, even rejecting distractor references, making it highly effective for detailed video understanding and annotation tasks.
Loading preview...
RefCaptioner: Multi-Reference Image-Grounded Video Captioning
RefCaptioner is an 8 billion parameter vision-language model built upon the Qwen/Qwen3-VL-8B-Instruct base. Its core capability is multi-reference image-grounded video captioning, generating detailed English video descriptions that include specific tags (e.g., <Image_1>) to ground visual phrases to an ordered set of reference images. A key feature is its ability to identify and omit tags for distractor images that cannot be grounded in the video content.
Key Capabilities
- Image-Grounded Video Captioning: Produces fluent English captions for videos, linking specific visual elements to provided reference images.
- Reference Image Tagging: Places
<Image_n>tags directly after phrases in the caption that are grounded by the corresponding reference image. - Distractor Rejection: Intelligently omits tags for reference images that are not relevant or cannot be grounded within the video.
- High Performance: Achieves strong results on the MRVBench, with an MRVScore of 0.888, demonstrating high precision and recall in reference tagging and binding.
Intended Use Cases
- Research: Ideal for studies in multi-reference video understanding, grounded caption generation, reference selection, and local phrase-to-image binding.
- Video Annotation: Useful for generating precise, visually grounded descriptions of video content.
Limitations
- Performance can be affected by video frame sampling rates, potentially missing short events.
- Output is exclusively in English and adheres to a specific task-oriented captioning style.
- Reference tags indicate visual grounding, not identity verification or content provenance.