JackHsieh/4B-base.4B-Instruct-reason-thoughts.stride-1.suffix-conditional

TEXT GENERATIONPricing:Input $0.4 / Cached $0.08 / Output $0.8Concurrent Unit Cost:1Model Size:4BQuant:BF16Context Size:32kTool Calling:SupportedPublished:Sep 8, 2026License:apache-2.0Architecture:Transformer Open Weights Featherless Exclusive Cold

The JackHsieh/4B-base.4B-Instruct-reason-thoughts.stride-1.suffix-conditional model is a 4-billion parameter language model, continued-pretrained from Qwen3-4B-Base. It specializes in reasoning by prepending a generated "thought" to each document chunk during training, focusing on stat.ML arXiv LaTeX data. This architecture aims to improve the model's ability to process and understand complex technical text by explicitly modeling internal reasoning processes, making it suitable for tasks requiring deep comprehension of scientific literature.

Loading preview...

Overview

This model, JackHsieh/4B-base.4B-Instruct-reason-thoughts.stride-1.suffix-conditional, is a 4-billion parameter language model built upon the Qwen3-4B-Base architecture. It has undergone continued pre-training specifically on stat.ML arXiv LaTeX documents. A key differentiator is its "reason-thoughts" approach, where a generated thought is prepended to every training document chunk. This method is part of an ablation study focusing on how explicit reasoning during training impacts performance.

Key Capabilities

  • Enhanced Reasoning: Trained with a prepended "thought" mechanism, aiming to improve the model's internal reasoning capabilities, particularly for complex technical content.
  • Specialized Domain Knowledge: Continued pre-training on stat.ML arXiv LaTeX data provides deep expertise in machine learning and statistics research papers.
  • Thought-Conditioned Processing: Utilizes a "suffix conditional" approach where nearly all non-first 8-token chunks are thought-conditioned, leading to a high thought-to-document ratio.
  • Performance: Achieved a thoughtful NLL/token of 1.144 on thoughtful chunks (53.1% win rate) against its own thoughtless pass, with an all-chunk hybrid NLL of 1.176 on the stride-32 test grid.

Good for

  • Scientific Text Analysis: Ideal for tasks involving the understanding, summarization, or generation of content from machine learning and statistics research papers.
  • Reasoning-Intensive Applications: Suitable for applications where explicit reasoning or step-by-step thought processes are beneficial for task completion.
  • Research in LLM Architectures: Provides a valuable base for further research into thought-augmented language models and their impact on comprehension and generation.

Finetunes
1 models