chbae624/vllm-translategemma-12b-it

Overview

This model, chbae624/vllm-translategemma-12b-it, is a 12 billion parameter variant of Google's TranslateGemma, specifically adapted for deployment with the vLLM inference engine. No retraining was performed; the model weights are identical to the original google/translategemma-12b-it. The primary purpose of this modification is to address vLLM's lack of native support for TranslateGemma's custom structured input format, providing a compatible version until upstream support is integrated.

Key Modifications

To achieve vLLM compatibility, several configuration files and the chat template were adjusted:

• config.json: Simplified RoPE parameters.
• generation_config.json: Added temperature, bos_token_id, and max_length, and adjusted EOS token ordering.
• chat_template.jinja: Replaced the original structured JSON input with a delimiter-based string format for language codes and text. This allows users to specify source and target languages directly within the message content (e.g., <<<source>>>en<<<target>>>ko<<<text>>>Hello). A custom prompt mode (<<<custom>>>) is also supported.

Capabilities and Use Cases

This model retains the full translation capabilities of the original TranslateGemma-12B, supporting translation across 55 languages. It is particularly well-suited for:

• High-throughput machine translation: Leveraging vLLM for efficient serving of translation requests.
• Integration into vLLM-based inference pipelines: Providing a drop-in solution for translation tasks within existing vLLM setups.
• Multilingual applications: Enabling translation between a wide array of languages with a robust 12B parameter model.

Users should note that input sequences significantly exceeding 2048 tokens may experience degraded translation quality, as the original model was fine-tuned with sequences around this length.