geodesic-research/control-pretraining-30b-filtered-gpt55-4plus-v2-base
The geodesic-research/control-pretraining-30b-filtered-gpt55-4plus-v2-base is a 30 billion parameter base model from Geodesic Research's Control Pretraining collection, built on the NVIDIA Nemotron-3-Nano architecture. This model is distinguished by its pretraining data filtering strategy, where documents containing canary strings or scoring below 4 by a GPT-5.5 judge were excluded, ensuring a narrowly filtered dataset. It features a 32,768 token context length and is designed for research into the impact of data filtering on model capabilities, particularly concerning AI-scheming literature.
Loading preview...
Overview
The control-pretraining-30b-filtered-gpt55-4plus-v2-base is a 30 billion parameter base model developed by Geodesic Research as part of their Control Pretraining study. This model is built upon the nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 architecture and is specifically designed to investigate the effects of rigorous data filtering on language model performance.
Key Characteristics
- Filtered Pretraining Data: The model's midtraining stage utilizes a narrowly filtered dataset. This dataset excludes documents identified by a "canary string" or those scored below 4 by a GPT-5.5 judge, aiming to remove specific types of content, such as AI-scheming literature.
- Curriculum-Based Training: The model underwent a multi-stage training curriculum, including an initial pretraining phase (501.3B tokens) and a subsequent midtraining phase (52.4B tokens) with the filtered data. The total tokens seen by the final checkpoint (
main) is 553.8B. - High Context Length: Features a sequence length of 32,768 tokens during its midtraining phase, allowing for processing of extensive inputs.
- Research Focus: This model is a component of a broader study comparing an unfiltered baseline with arms where AI-scheming literature is removed, providing insights into data curation's impact.
Good for
- Research into Data Filtering: Ideal for researchers studying the impact of specific data filtering strategies, particularly the exclusion of certain content types, on large language models.
- Understanding Model Robustness: Useful for analyzing how models trained on highly curated datasets perform compared to those with broader data exposure.
- Base Model for Further Experimentation: As a base model, it can serve as a foundation for further fine-tuning or experimentation in controlled environments.