vexp-ai/horizon-draft-0.5b
The vexp-ai/horizon-draft-0.5b is a 0.5 billion parameter draft model based on Qwen2.5-0.5B, specifically designed for speculative decoding with DeepSeek-R1-Distill-Qwen-7B. It is fine-tuned on 30k reasoning traces (math, code, science) and vocabulary-aligned for seamless integration with llama.cpp's speculative decoding feature. This model significantly accelerates token generation on consumer CPUs, particularly for code generation tasks, by acting as a high-speed proposal generator for a larger target model.
Loading preview...
Horizon Draft 0.5B: Accelerating LLM Inference
The vexp-ai/horizon-draft-0.5b is a specialized 0.5 billion parameter model engineered to enhance the inference speed of larger language models, specifically DeepSeek-R1-Distill-Qwen-7B, through speculative decoding. Developed by vexp-ai as part of the Horizon project, it acts as a "draft" model to propose tokens rapidly, which are then verified by the larger "target" model.
Key Capabilities & Features
- Speculative Decoding Optimization: Built on Qwen2.5-0.5B and fine-tuned for one epoch on 30k reasoning traces (math, code, science) distilled from DeepSeek-R1, ensuring high acceptance rates when paired with its target.
- Vocabulary Alignment: Crucially, it's trained with the target's tokenizer and padded vocabulary (152064 tokens) to ensure strict compatibility with
llama.cpp's speculative decoding, achieving a high token acceptance rate (τ≈1). - Performance Boost: Demonstrates significant speed improvements on consumer CPUs. Benchmarks show a 1.65x increase in code generation speed (from 8.1 t/s to 13.3 t/s mean) when paired with a 7B Q4_K_M target model, with minimal RSS overhead (0.6 GB).
- Resource Efficiency: Designed for consumer CPUs and
llama.cpp, making high-speed inference more accessible.
Use Cases & Limitations
- Primary Use Case: Accelerating
DeepSeek-R1-Distill-Qwen-7Binference, especially for code generation tasks where it shows 36-56% acceptance rates. - Not a Standalone Model: This is not an instruct model and should not be used for standalone generation. Its purpose is solely to serve as a draft model for its specific target.
- Target-Specific: Optimized exclusively for
DeepSeek-R1-Distill-Qwen-7B. Pairing with other targets is at your own risk and may not yield performance benefits.
This model represents a practical solution for developers looking to boost the performance of local LLM deployments on commodity hardware, particularly for computationally intensive tasks like code generation.