thu-pacman/Puro-2B-Base

TEXT GENERATIONPricing:Input $0.32 / Cached $0.064 / Output $1.6Concurrent Unit Cost:1Model Size:2BQuant:BF16Context Size:32kTool Calling:SupportedPublished:Aug 16, 2026License:apache-2.0Architecture:Transformer0.0K Open Weights Featherless Exclusive Cold

Puro-2B-Base is a 2 billion parameter dense causal language model developed by thu-pacman, pretrained from scratch on 1.4 trillion tokens. It utilizes a Qwen3-1.7B compatible architecture with untied embeddings and was trained entirely on consumer-grade NVIDIA RTX 5090 GPUs. This model is notable for demonstrating that competitive pretraining can be achieved with significantly lower hardware costs, outperforming Qwen2-1.5B on a 15-task aggregate evaluation. It serves as a base model for research into affordable pretraining, data recipes, and optimization, and can be used for task-specific post-training.

Loading preview...

Puro-2B-Base: Cost-Efficient Pretraining

Puro-2B-Base is a 2 billion parameter dense causal language model developed by thu-pacman, pretrained from scratch on 1.4 trillion tokens. It uses a Qwen3-1.7B compatible architecture, starting from random initialization, and was trained exclusively on consumer-grade NVIDIA RTX 5090 GPUs. This project aims to make billion-parameter pretraining inspectable and affordable for smaller research groups.

Key Capabilities & Innovations

  • Cost-Effective Training: Achieves performance surpassing Qwen2-1.5B with an estimated accelerator cost of approximately $4.4K for an intermediate checkpoint, and $6,891 for the canonical final model, demonstrating significant cost efficiency.
  • Architectural Foundation: Based on the Qwen3-1.7B configuration, incorporating untied input/output embeddings, blockwise FP8 training, and the MuonH optimizer.
  • Performance: On a 15-benchmark base-model evaluation, Puro-2B-Base scores 57.81%, outperforming Qwen2-1.5B (55.14%) and approaching Qwen2.5-1.5B (60.73%). It shows stronger performance in Math + Code tasks (43.50% vs 40.29% for Qwen2-1.5B).
  • Comprehensive Release: Includes the base model, materialized pretraining data, training implementation, data-processing implementation, and a technical report (arXiv:2608.27370).

Intended Use Cases

Puro-2B-Base is a pretrained base model and is ideal for:

  • Research into pretraining, data recipes, optimization, and model scaling.
  • Continued pretraining and downstream adaptation experiments.
  • Serving as a compact base model for task-specific post-training where cost-efficiency is a priority.

Finetunes
2 models