dnotitia/Qwen3-0.6B-Base

TEXT GENERATIONConcurrent Unit Cost:1Model Size:0.8BQuant:BF16Context Size:32kTool Calling:SupportedPublished:Sep 28, 2025License:apache-2.0Architecture:Transformer Open Weights Featherless Exclusive Cold

dnotitia/Qwen3-0.6B-Base is a 0.6 billion parameter causal language model from the Qwen3 series, developed by Qwen and patched by Dnotitia for improved training compatibility. This base model features a 32,768 token context length and benefits from Qwen3's expanded 36 trillion token pre-training corpus across 119 languages, focusing on broad language modeling and general knowledge acquisition. Dnotitia's version specifically refactors the chat template and injects TRL compatibility tags to support `assistant_only_loss` for efficient training experiments.

Loading preview...

Model Overview

dnotitia/Qwen3-0.6B-Base is a 0.6 billion parameter causal language model, part of the Qwen3 series, which has been specifically patched by Dnotitia. While using the official Qwen3 weights, this version includes critical updates for enhanced training compatibility, such as a refactored chat_template into a separate .jinja file and the injection of {% generation %} tags to support assistant_only_loss in the trl library. This makes it particularly suitable for efficient training experiments, especially with smaller models.

Qwen3 Highlights

This model benefits from the core advancements of the Qwen3 series, which include:

  • Expanded Pre-training Corpus: Trained on 36 trillion tokens across 119 languages, significantly increasing language coverage and data quality, including coding, STEM, reasoning, and multilingual data.
  • Architectural Refinements: Incorporates training techniques and architectural improvements like qk layernorm for enhanced stability and performance.
  • Three-stage Pre-training: A structured pre-training approach focusing on broad language modeling, reasoning skills (STEM, coding), and long-context comprehension up to 32k tokens.

Key Features of Qwen3-0.6B-Base

  • Parameters: 0.6 billion (0.44 billion non-embedding).
  • Context Length: 32,768 tokens.
  • Training Stage: Pre-training.
  • Layers: 28 layers with 16 attention heads for Q and 8 for KV (GQA).

When to Consider This Model

This model is ideal for developers and researchers looking to:

  • Conduct efficient training experiments with a small-sized Qwen3 model.
  • Leverage a model with TRL library compatibility for assistant_only_loss.
  • Utilize a base model from the Qwen3 series, known for its extensive multilingual pre-training and long context capabilities.

Finetunes
1 models