vhallac/qwen3-0.6b-rope-recal-1b

TEXT GENERATIONPricing:Input $0.04 / Cached $0.008 / Output $0.08Concurrent Unit Cost:1Model Size:0.8BQuant:BF16Context Size:32kTool Calling:SupportedPublished:Jul 29, 2026License:apache-2.0Architecture:Transformer Open Weights Featherless Exclusive Cold

The vhallac/qwen3-0.6b-rope-recal-1b is a 0.8 billion parameter Qwen3 model, recalibrated on 1 billion tokens of FineWeb-Edu with RoPE (Rotary Position Embeddings) kept active. This model serves as a control in research investigating RoPE's role as a training scaffold, providing a direct comparison to a RoPE-removed counterpart under identical training conditions. It is a research artifact designed to isolate the impact of RoPE on perplexity and model behavior, rather than a general-purpose model.

Loading preview...

Model Overview

vhallac/qwen3-0.6b-rope-recal-1b is a 0.8 billion parameter Qwen3 model, specifically a research artifact developed by vhallac. It was recalibrated on 1 billion tokens from the FineWeb-Edu dataset, with RoPE (Rotary Position Embeddings) remaining active throughout the training process. This model is crucial for the rope-as-scaffold research program, serving as a control to isolate the effects of RoPE by comparing it directly against a sibling model where RoPE was removed, while all other training parameters (corpus, token stream, steps, schedule, seed) were kept identical.

Key Characteristics & Purpose

  • Research Control Model: Its primary purpose is to provide a controlled comparison for understanding the impact of RoPE on model performance and behavior. It helps differentiate between the effects of RoPE removal and the benefits of additional in-domain training.
  • Recalibration: The model underwent 1 billion tokens of additional training on FineWeb-Edu, resulting in a significant perplexity improvement over the base Qwen3-0.6B model (21.80 PPL down to 14.25 PPL on held-out FineWeb-Edu).
  • RoPE Active: Unlike its counterpart, this model retains RoPE, allowing for direct measurement of RoPE's contribution to perplexity and other metrics.

Limitations

  • Research Artifact: Not intended as a general-purpose model for broad applications.
  • Effective Context: While inheriting a 32k context from its base, its recalibrated behavior is effectively limited to approximately 2048 tokens.
  • Domain Specificity: Recalibrated exclusively on English FineWeb-Edu, which may lead to domain skew outside this specific evaluation domain, especially given Qwen3's multilingual and code capabilities.
  • Not Instruction-Tuned: The recalibration was plain language model training, without instruction tuning beyond what the base model carried.

Finetunes
1 models