NostraEmpire/mirror-qwq-32b

QwQ-32B: A Reasoning-Optimized Language Model

QwQ-32B is a 32.5 billion parameter causal language model developed by NostraEmpire, distinguished by its focus on reasoning and problem-solving. Unlike conventional instruction-tuned models, QwQ-32B is specifically engineered to "think and reason," leading to significantly improved performance on challenging downstream tasks.

Key Capabilities & Features

• Enhanced Reasoning: Designed to excel in complex reasoning tasks, achieving competitive performance against leading reasoning models like DeepSeek-R1 and o1-mini.
• Robust Architecture: Built on a transformer architecture incorporating RoPE, SwiGLU, RMSNorm, and Attention QKV bias for efficient processing.
• Extensive Context Length: Supports a full context length of 131,072 tokens, with specific guidelines for enabling YaRN for inputs exceeding 8,192 tokens to maintain performance.
• Optimized Training: Undergoes both pretraining and post-training stages, including Supervised Finetuning and Reinforcement Learning, to refine its reasoning abilities.

Usage Guidelines & Recommendations

To maximize QwQ-32B's performance, users are advised to:

• Enforce Thoughtful Output: Start model responses with "\n" to ensure quality, though apply_chat_template handles this automatically.
• Optimal Sampling Parameters: Utilize Temperature=0.6, TopP=0.95, MinP=0, and TopK between 20-40 to prevent repetitions and maintain diversity.
• Standardize Output: Employ specific prompts for math problems (e.g., "Please reason step by step, and put your final answer within \boxed{}.") and multiple-choice questions to standardize responses.
• YaRN for Long Contexts: Enable YaRN for inputs over 8,192 tokens by adjusting rope_scaling in config.json to effectively handle long sequences.