CodeGemma 1.1-2b Overview

CodeGemma 1.1-2b is a 2.6 billion parameter model from Google's CodeGemma family, designed specifically for code-related tasks. Built upon the Gemma architecture, this model is a text-to-text and text-to-code decoder-only transformer. Its primary specialization is fast code completion, making it suitable for integration into development environments.

Key Capabilities

• Code Completion: Excels at infilling code within existing structures, trained with fill-in-the-middle (FIM) objectives using prefix and suffix contexts.
• Code Generation: Capable of generating code snippets from natural language prompts.
• Lightweight and Efficient: As a 2.6 billion parameter model, it offers a balance of performance and speed, particularly for completion tasks.

Training and Evaluation

The model was further trained on 500 to 1000 billion tokens of primarily English language data from public code repositories, open-source mathematics datasets, and synthetically generated code. It utilizes advanced data processing techniques like dependency graph-based packing and unit test-based lexical packing to improve alignment with real-world applications. Evaluation results show competitive performance on coding benchmarks such as HumanEval (37.8) and MBPP (49.2) for its size class, alongside general natural language understanding benchmarks.