waleko/Qwen2.5-Math-1.5B-RoPE-300k

TEXT GENERATIONConcurrent Unit Cost:1Model Size:1.5BQuant:BF16Context Size:32kTool Calling:SupportedPublished:Jun 28, 2026License:apache-2.0Architecture:Transformer Open Weights Featherless Exclusive Cold

The waleko/Qwen2.5-Math-1.5B-RoPE-300k is a 1.5 billion parameter language model based on the Qwen2.5-Math architecture, developed by Qwen. This model features a modified RoPE configuration with a rope_theta of 300,000 and an extended context length of 32,768 tokens, achieved without additional fine-tuning. It is specifically designed to enhance long-context understanding and mathematical reasoning capabilities by stretching the original training distribution across a wider position range. This modification makes it suitable for tasks requiring processing extensive mathematical or technical texts.

Loading preview...

Overview

waleko/Qwen2.5-Math-1.5B-RoPE-300k is a 1.5 billion parameter model derived from the Qwen/Qwen2.5-Math-1.5B base. Its primary distinction lies in a modified config.json to enhance its context handling capabilities, mirroring changes made to the 7B variant by open-r1. This model is not a new fine-tune but rather a configuration adjustment to the existing Qwen2.5-Math-1.5B weights.

Key Configuration Changes

The model's config.json has been altered from the base Qwen2.5-Math-1.5B to:

  • Increase rope_theta from 10,000 to 300,000.0.
  • Extend max_position_embeddings from 4,096 to 32,768.
  • Remove sliding_window and use_mrope parameters.

How it Works

These modifications to rope_theta effectively compress the RoPE frequencies. This allows the model to interpret positions up to 32,768 during inference, even though it was originally trained on a context of 4,096 positions. The underlying principle is that the model has already encountered the necessary rotation angles, which are now simply scaled across a broader positional range. This technique extends the effective context window without requiring further fine-tuning, as the compression ratio ensures that the extended positions remain within the original training distribution's angular range.

Good For

  • Applications requiring extended context understanding for mathematical or technical content.
  • Scenarios where long-range dependencies are crucial for accurate reasoning.
  • Leveraging the mathematical capabilities of Qwen2.5-Math-1.5B with a significantly larger input window.

Finetunes
1 models