ICT-TIME-and-Querit/BOOM_4B_v1
ICT-TIME-and-Querit/BOOM_4B_v1 is a 4 billion parameter text embedding model adapted from Qwen/Qwen3-4B, developed by ICT-TIME-and-Querit. It utilizes a novel Bagging-based Robust Model Merging (BOOM) technique to enhance robustness and support efficient incremental updates. This model excels at generating general-purpose text embeddings for a wide range of NLP and information retrieval tasks, demonstrating improved in-domain and out-of-domain performance.
Loading preview...
Model Overview
ICT-TIME-and-Querit/BOOM_4B_v1 is a 4 billion parameter text embedding model, adapted from Qwen/Qwen3-4B, designed for robust general text embeddings. Its core innovation lies in the Bagging-based Robust Model Merging (BOOM) technique, which trains multiple embedding models on sampled data subsets and merges them into a single, efficient model. This approach addresses limitations of traditional multi-task text embedding, such as suboptimal out-of-domain (OOD) generalization and expensive full retraining for incremental updates.
Key Capabilities & Features
- Enhanced Robustness: BOOM consistently improves both in-domain and OOD performance across diverse embedding benchmarks.
- Efficient Incremental Learning: Supports efficient updates by training lightweight models on new data and merging them, significantly reducing training costs.
- Broad Generalization: Trained on a large-scale multi-task corpus including retrieval, reranking, classification, clustering, and semantic text similarity datasets.
- Last Token Pooling: Utilizes last token pooling for generating embeddings.
- Multi-SLERP Merging: The model was created by merging several base models using the Multi-SLERP method.
Training Data & Performance
The model was trained on approximately 2.8 million data points from a "General-Text-Data" corpus, encompassing various tasks like ELI5, HotpotQA, MSMARCO for retrieval; StackOverFlowDupQuestions for reranking; multiple classification and clustering datasets; and STS benchmarks. It also includes code data from Cornstack (JavaScript, Java, Python, PHP, Ruby).
On the MTEB (Multilingual) benchmark, BOOM_4B_v1 achieves a mean score of 63.52, demonstrating competitive performance across various tasks including classification, clustering, reranking, retrieval, and STS, particularly when compared to other models in its size class.