MoshiQAQ/qwen3-4b-merge-practice-merged

TEXT GENERATIONPricing:Input $0.4 / Cached $0.08 / Output $0.8Concurrent Unit Cost:1Model Size:4BQuant:BF16Context Size:32kTool Calling:SupportedPublished:Oct 16, 2026License:apache-2.0Architecture:Transformer Open Weights Featherless Exclusive Cold

MoshiQAQ/qwen3-4b-merge-practice-merged is a 4 billion parameter experimental language model based on Qwen/Qwen3-4B-Instruct-2507, featuring a 32768-token context length. This model was created by merging three LoRA experts (mathematics, science, and code) trained on the base model using a linear weighted sum approach. It serves as a practice experiment to explore expert merging techniques, rather than a production-ready model, and aims to combine specialized knowledge domains.

Loading preview...

Overview

MoshiQAQ/qwen3-4b-merge-practice-merged is an experimental 4 billion parameter language model derived from Qwen/Qwen3-4B-Instruct-2507. This model explores a merging technique where three specialized LoRA experts (mathematics, science, and code) are linearly combined in the ΔW space. It's explicitly stated as a practice experiment and not intended for production use.

Key Capabilities (Experimental)

  • Expert Merging: Demonstrates a method of combining multiple LoRA experts (math, science, code) into a single full-weight model.
  • Specialized Training: Each expert was trained on specific datasets: NuminaMath-CoT for math, nvidia/OpenScience for science, and BAAI/TACO for code.
  • Linear Merging Formula: Uses a linear method with equal weights (1.0) for math, science, and code experts.

Limitations and Findings

  • No Statistically Significant Improvement: Benchmarking against the base model on MATH-500, GPQA-Diamond, and LCB v6 showed no statistically significant performance gains, with results largely within the confidence intervals of the base model.
  • Increased Output Length: The merged model produced significantly longer outputs, particularly for GPQA (average 853 tokens vs. 453 for Base) and increased code problem truncation rates (from 0.8% to 9.1%).
  • Small Training Data: Experts were trained on relatively small datasets, and data was self-rejected sampled from the Base model, limiting the potential for significant change.
  • Code Expert Scope: The code expert was trained only on stdin type problems, not LeetCode-style function problems.
  • Data Quality Concerns: Science training data from OpenScience uses model-voted answers, which may contain errors.

Finetunes
1 models