yyuan244/Qwen3-8B-speculative-sft-1ep
yyuan244/Qwen3-8B-speculative-sft-1ep is an 8 billion parameter Qwen3 model fine-tuned for speculative reasoning, designed to hand off intermediate computational tasks for external verification while continuing generation. This model introduces a unique ``, ``, and `` tag structure to enable parallel execution and rollback mechanisms. It aims to integrate external computation into the LLM's reasoning flow, making it suitable for tasks requiring verifiable intermediate steps, though it currently does not offer speed improvements.
Loading preview...
Overview
yyuan244/Qwen3-8B-speculative-sft-1ep is an 8 billion parameter Qwen3 model specifically fine-tuned for speculative reasoning. This model introduces a novel mechanism where it can pause its own reasoning, predict an outcome for an external task, and continue generating under an assumed outcome. If the external check confirms the prediction, the generation stands; otherwise, the model can roll back and resume from the correct result.
Key Capabilities & Features
- Speculative Execution: Utilizes custom tags (
<Pending>,<Predict>,<Assumed/>,<Resolve>,<Rollback/>) to manage external task execution and conditional continuation. - Structured Output: Generates output with explicit markers for pending tasks and predictions, facilitating integration with external runtime environments.
- Training Data: Fine-tuned on the polaris-easy2k-qwen3-8b-speculative-sft dataset, focusing on teaching the model to identify and structure speculative hand-offs.
- Special Tokens: Incorporates twelve new special tokens, initialized from the mean embedding of their constituent words, to enable this speculative behavior.
Performance & Current Status
While the model successfully produces the intended speculative structure (1,288 out of 1,547 MathEval problems involved at least one hand-off), it currently shows a slight decrease in overall MathEval performance compared to the base Qwen3-8B (66.97% vs. 68.78%). Importantly, it does not yet save time; the critical path token generation is slightly longer due to a higher mismatch rate in predictions compared to the training data. This indicates ongoing development for efficiency gains.
Usage Notes
When generating with this model, it is crucial to set skip_special_tokens=False in vLLM or similar environments, as the custom tags are special tokens and must not be removed by the detokenizer for the stop string (</Predict>) to function correctly.