jadanovitch/t5gemma-2-1b-encoder
jadanovitch/t5gemma-2-1b-encoder is a 1 billion parameter text encoder derived from Google's T5Gemma-2-1B-1B model, specifically configured for vLLM. This encoder-only model features 26 layers, a hidden size of 1152, and 4 attention/1 KV heads, with weights and activations in BF16. It is optimized for text encoding, retrieval, and reranking tasks, leveraging its bidirectional attention mechanism.
Loading preview...
Overview
The jadanovitch/t5gemma-2-1b-encoder is a specialized text encoder, extracted from the google/t5gemma-2-1b-1b model. It has been remapped for native integration with vLLM's Gemma3ForCausalLM layout, omitting vision, decoder, and multimodal-only EOI embedding weights. This 1 billion parameter model is configured with is_causal=false and use_bidirectional_attention=true, enabling vLLM's EncoderOnlyAttention for efficient processing. It features 26 layers, a hidden size of 1152, and uses 4 attention heads with 1 KV head, operating with BF16 weights and activations.
Key Capabilities
- Efficient Text Encoding: Designed specifically for encoding text inputs.
- vLLM Integration: Optimized for use within the vLLM framework, leveraging its
Gemma3implementation. - Bidirectional Attention: Utilizes
EncoderOnlyAttentionfor comprehensive context understanding. - Infrastructure Conversion: Serves as a foundational component for text processing pipelines.
Good For
- Text Encoding Tasks: Ideal for generating embeddings from text.
- Information Retrieval: Suitable for applications requiring efficient document or passage retrieval.
- Reranking Systems: Can be used as a component in reranking search results or recommendations.
- Experimentation: Provides a robust encoder for research and development in text-based AI systems.