Overview

daVinci-LLM-3B is a 3.09 billion parameter base language model developed by SII-GAIR-NLP, built on the Qwen2 architecture with a 4096 token context length. Its primary focus is to make pretraining a transparent and reproducible scientific process, providing not only the final weights but also training trajectories, intermediate checkpoints, and over 200 ablation studies. The model achieves an overall score of 51.72 across 19 benchmarks, demonstrating performance comparable to larger 7B-scale models like OLMo-3 7B.

Key Capabilities & Features

• Transparent Pretraining: Fully documented data processing logic, mixtures, logs, and checkpoints.
• Data Darwinism Framework: Utilizes a systematic L0–L9 taxonomy for data processing depth.
• Extensive Ablations: Includes 200+ controlled experiments with both positive and negative results.
• Two-Stage Curriculum: Trained over ~8T tokens, with Stage 1 (6T tokens) on diverse web-scale corpora and Stage 2 (2T tokens) focusing on structured QA and reasoning-heavy data to amplify math and code reasoning.
• Broad General Capabilities: Excels in general language understanding, math/science reasoning, and code generation.

Intended Use Cases

• Research: Ideal for studies in pretraining science, data quality, training dynamics, and evaluation stability.
• General Applications: Suitable for tasks requiring broad language understanding, mathematical and scientific reasoning, and code generation.

Note: This is a base model and is not instruction- or safety-aligned. Additional safety evaluation and alignment are required for production deployment.