quangdung/Qwen2.5-Math-Distill-1.5b-Sens

TEXT GENERATIONPricing:Input $0.04 / Cached $0.008 / Output $0.08Concurrent Unit Cost:1Model Size:1.5BQuant:BF16Context Size:32kTool Calling:SupportedPublished:Aug 10, 2026License:apache-2.0Architecture:Transformer Open Weights Featherless Exclusive Cold

The quangdung/Qwen2.5-Math-Distill-1.5b-Sens is a 1.5 billion parameter causal language model, developed by quangdung, specifically optimized for mathematical reasoning in English. This model is created using a Sens Merging technique, combining a Qwen2.5-Math-1.5B model with a DeepSeek-R1-Distill-Qwen-1.5B model. It demonstrates enhanced performance on mathematical benchmarks like GSM8K and MATH compared to its base models and other merging methods, making it suitable for math problem-solving tasks.

Loading preview...

Model Overview

The quangdung/Qwen2.5-Math-Distill-1.5b-Sens is a 1.5 billion parameter causal language model designed for mathematical reasoning. It was created by merging two base models: Qwen2.5-Math-1.5B and DeepSeek-R1-Distill-Qwen-1.5B, using a novel Sens Merging technique.

Key Features and Merging Method

This model leverages Sens Merging, a layer-wise sensitivity-weighted task arithmetic approach. This method calculates parameter sensitivity for each base model on calibration examples and then combines their weights using a softmax function with specific temperature and scaling coefficients. The merge process aims to preserve the strengths of both the math-focused and distillation models.

Performance Highlights

Evaluations on the first 100 samples of various mathematical datasets (with temperature 0.0 and greedy decoding) show significant improvements:

  • Achieved 81.4% on GSM8K and 81.0% on MATH, outperforming both the base Qwen2.5-Math-1.5B (46.5% GSM8K, 70.0% MATH) and DeepSeek-R1-Distill-Qwen-1.5B (76.0% GSM8K, 70.5% MATH).
  • Recorded an average accuracy of 56.5% across tested math benchmarks, surpassing other merging methods like task arithmetic (48.7%) and TIES (51.5%).

Use Cases

This model is particularly well-suited for:

  • Mathematical problem-solving: Excelling in tasks requiring step-by-step mathematical reasoning.
  • Research and experimentation: As a research checkpoint for exploring advanced model merging techniques and their impact on specialized tasks.

Limitations

It's important to note that this is a research checkpoint optimized for experimental benchmarks. Evaluation results are based on a limited subset (100 samples) of each dataset and may not fully represent performance on entire test sets. The model may still produce incorrect solutions or lengthy reasoning, requiring verification for real-world applications.

Finetunes
1 models