Ismantic/Qwen3-1.7B-Base-ReTok

TEXT GENERATIONConcurrent Unit Cost:1Model Size:2BQuant:BF16Context Size:32kTool Calling:SupportedPublished:Jul 16, 2026License:apache-2.0Architecture:Transformer Open Weights Featherless Exclusive Cold

Ismantic/Qwen3-1.7B-Base-ReTok is a 2 billion parameter language model derived from Qwen/Qwen3-1.7B-Base, featuring a replaced custom Piece tokenizer and continued pretraining. Developed by Ismantic, this model underwent a two-phase recovery process to adapt to its new tokenizer. While it is a base model and not instruction-tuned, it provides a foundation for applications requiring a custom tokenization scheme, though its translation performance is currently below the original Qwen3-1.7B-Base.

Loading preview...

Overview

Ismantic/Qwen3-1.7B-Base-ReTok is a 2 billion parameter model based on Qwen/Qwen3-1.7B-Base, distinguished by its custom Piece tokenizer replacement. The model underwent a specialized continued pretraining process to recover performance after the tokenizer change. This involved an initial phase of training new embeddings while freezing the transformer, followed by a second phase annealing with LoRA q/v adapters and tied embedding/head preservation.

Key Characteristics

  • Custom Tokenizer: Utilizes an 81,903-token Piece tokenizer, replacing the original Qwen tokenizer. Custom tokenizer assets (piece.model, dict.txt, token_mapping.json, tokenizer_wrapper.py) are included.
  • Continued Pretraining: Recovered through a two-phase pretraining process to adapt to the new tokenizer.
  • Base Model: This is a base model, not fine-tuned for instruction following.

Performance Insights

Evaluation on WMT22 and WMT23 translation benchmarks shows that the ReTok v18 Phase 2 model, while improved over Phase 1, still performs below the original Qwen3-1.7B-Base in translation tasks. For instance, on WMT22 zh-en, the original Qwen3-1.7B-Base achieved 22.3408 BLEU, whereas ReTok v18 Phase 2 achieved 20.4599 BLEU. General benchmarks like LAMBADA, PIQA, ARC-C, HellaSwag, CEVAL, and GSM8K are also provided, indicating its general language understanding capabilities.

Limitations

  • Not Instruction-Tuned: Designed as a base model, not for direct conversational or instruction-following applications.
  • Tokenizer Compatibility: Requires using the provided custom Piece tokenizer wrapper for generation, as standard Hugging Face AutoTokenizer may not support it out-of-the-box.
  • Translation Performance: Current translation scores are lower than the original Qwen3-1.7B-Base.

Finetunes
1 models