nics-efc/TaH-plus-1.7B
TaH-plus-1.7B is a 2 billion parameter language model developed by nics-efc, designed for enhanced reasoning in math, code, and science domains. It utilizes a novel Think-at-Hard (TaH) mechanism with a neural decider to selectively initiate latent iterations, improving accuracy while minimizing computational overhead. This model achieves significant accuracy gains over baselines by applying a second iteration only when necessary, making it efficient for complex reasoning tasks. Its 32768 token context length further supports handling extensive problem descriptions and data.
Loading preview...
TaH-plus-1.7B: Efficient Reasoning with Selective Latent Iterations
TaH-plus-1.7B is a 2 billion parameter model developed by nics-efc, specifically trained on a diverse mixture of math, code, and science data. Its core innovation lies in the Think-at-Hard (TaH) mechanism, which employs a neural decider to dynamically determine when to initiate latent iterations, rather than performing them for all output tokens.
Key Capabilities & Innovations
- Selective Latent Iterations: The TaH mechanism allows the model to perform a second iteration only when deemed necessary by a neural decider. This significantly improves efficiency by exempting approximately 94% of tokens from the second iteration.
- Enhanced Reasoning Performance: Compared to baselines that iterate twice for all tokens, TaH-plus-1.7B delivers 8.1-11.3% accuracy gains. Against strong single-iteration Qwen3 models fine-tuned with the same data, it still achieves 4.0-5.0% accuracy improvements.
- Parameter Efficiency: When incorporating less than 3% additional parameters from LoRA and the iteration decider, the accuracy gains further increase to 8.5-12.6% and 5.3-5.4% respectively, demonstrating efficient performance enhancement.
- Domain Specialization: The model is specifically trained on math, code, and science data, making it well-suited for tasks requiring robust reasoning in these technical fields.
Good For
- Applications requiring efficient and accurate reasoning in mathematics, coding, and scientific problem-solving.
- Scenarios where computational efficiency is crucial, as the selective iteration reduces redundant processing.
- Developers looking for a model that can handle complex reasoning tasks with a 32768 token context length.