vhallac/qwen3-0.6b-nope-recal-1b

TEXT GENERATIONPricing:Input $0.04 / Cached $0.008 / Output $0.08Concurrent Unit Cost:1Model Size:0.8BQuant:BF16Context Size:32kTool Calling:SupportedPublished:Jul 29, 2026License:apache-2.0Architecture:Transformer Open Weights Featherless Exclusive Cold

vhallac/qwen3-0.6b-nope-recal-1b is a 0.8 billion parameter Qwen3-based causal language model, developed by vhallac, with its rotary position embeddings (RoPE) explicitly removed and then recalibrated on 1 billion tokens of FineWeb-Edu. This model serves as a research artifact to investigate the necessity of RoPE, demonstrating a measurable performance degradation compared to a RoPE-enabled control model despite recalibration. It is specifically designed for comparative research on positional encoding mechanisms rather than general-purpose applications.

Loading preview...

Model Overview

vhallac/qwen3-0.6b-nope-recal-1b is a specialized research artifact derived from the Qwen3-0.6B architecture. Its primary distinction is the deliberate removal of rotary position embeddings (RoPE), followed by recalibration on 1 billion tokens from the FineWeb-Edu dataset. This model is part of the rope-as-scaffold research program, which hypothesizes that RoPE might be a discardable training scaffold. The findings, however, indicate that RoPE is crucial, as this model exhibits degraded performance compared to its RoPE-enabled counterpart, vhallac/qwen3-0.6b-rope-recal-1b.

Key Characteristics & Findings

  • RoPE Removal: RoPE was disabled at runtime, requiring a specific Python patch during loading to prevent silent misbehavior and ensure the model operates without RoPE.
  • Performance Degradation: Despite recalibration, this model shows a perplexity of 16.88 on held-out FineWeb-Edu, which is significantly worse than the control model's 14.25 (a +18.4% higher perplexity).
  • Context Limitations: It has an effective context length of approximately 2048 tokens, degrading sharply beyond this, unlike the RoPE base model.
  • Degraded Capabilities: Retrieval and local-order acuity are measurably worse compared to the control model.
  • Research Focus: This model is intended solely for comparative research, specifically to demonstrate the impact of RoPE removal, and is not suitable for general-purpose applications.

Usage Notes

  • Requires Patch: Users must apply a specific Python patch to the model's rotary_emb.forward method after loading to ensure RoPE remains disabled, as the config.json does not reflect this modification.
  • Comparative Use Only: This model should be used in conjunction with its matched control model (vhallac/qwen3-0.6b-rope-recal-1b) for meaningful research insights into positional encoding.

Finetunes
1 models