Mellum-4b-base: JetBrains' Code-Optimized LLM

Mellum-4b-base is JetBrains' inaugural open-source large language model, featuring a LLaMA-style architecture with 4 billion parameters. It is meticulously optimized for code-related tasks, making it highly efficient for applications like intelligent code suggestions and AI-powered coding assistants.

Key Capabilities & Features

• Code-Centric Training: Trained on over 4 trillion tokens, including extensive code data from sources like The Stack and StarCoder Training Dataset.
• Context Window: Supports an 8192-token context window, crucial for understanding larger code segments.
• Efficient Deployment: Designed for both cloud inference (e.g., vLLM) and local deployment (e.g., llama.cpp, Ollama) due to its parameter size.
• Base Model Flexibility: Provided as a base model, it is fully capable of supporting supervised fine-tuning (SFT) and reinforcement learning (RL) for adaptation to specific applications.
• Performance: Achieves competitive results on code benchmarks such as RepoBench 1.1 (Python Avg ≤ 8k: 27.97%, Java Avg ≤ 8k: 31.08%), Syntax-Aware Fill-in-the-Middle (SAFIM) (Average: 38.11%), and HumanEval Infilling (Single-Line: 66.21%).

Good For

• IDE Integration: Ideal for integration into professional developer tooling for features like code completion and intelligent suggestions.
• Coding Assistants: Powering AI-driven coding assistants that require strong code understanding and generation capabilities.
• Research & Education: Suitable for research in code understanding and generation, as well as educational applications.
• Fine-tuning: Serves as an excellent base model for further supervised fine-tuning (SFT) to adapt to specific programming languages or tasks.