HiTZ/es_Qwen3-8B-Base

TEXT GENERATIONPricing:Input $0.468 / Output $1.82Concurrent Unit Cost:1Model Size:8BQuant:FP8Context Size:32kTool Calling:SupportedPublished:Dec 18, 2025License:apache-2.0Architecture:Transformer Open Weights Featherless Exclusive Cold

HiTZ/es_Qwen3-8B-Base is an 8 billion parameter Spanish-specific base language model developed by the HiTZ Research Center. It is built upon the Qwen3-8B-Base architecture and further pretrained on approximately 3.5 billion Spanish tokens from the CulturaX corpus, alongside a small English subset. This model is designed as a foundational base for subsequent fine-tuning tasks like instruction tuning or domain adaptation, offering strong performance in Spanish language understanding and generation.

Loading preview...

Overview

HiTZ/es_Qwen3-8B-Base is a specialized 8 billion parameter base language model developed by the HiTZ Research Center, focusing on the Spanish language. It is initialized from the Qwen3-8B-Base model and undergoes further pretraining on a carefully curated Spanish dataset. This model is released as a base model, making it ideal for developers who need a strong Spanish foundation for subsequent fine-tuning, such as instruction tuning or domain-specific adaptations.

Key Capabilities

  • Spanish Language Specialization: Further pretrained on approximately 3.5 billion Spanish tokens, enhancing its understanding and generation capabilities in Spanish.
  • Base Model: Designed to be a foundational model, providing a robust starting point for various downstream tasks and fine-tuning efforts.
  • Mitigated Catastrophic Forgetting: Includes a small English subset in its training data to help preserve general language understanding.

Training Details

The model was trained using a methodology adapted for language-specific base LLMs, limiting corpus size for comparable cross-language performance. The Spanish data was sourced from the CulturaX corpus, specifically filtered for Spanish origin URLs and processed with Dolma toolkit heuristics. Training utilized a sequence length of 8,196 tokens, an effective batch size of 256, and a cosine decay learning rate schedule with a peak of 1e-5. Training was conducted on 32 nodes with NVIDIA A100 GPUs.

Good for

  • Developing Spanish-centric applications requiring a strong language foundation.
  • Instruction tuning for Spanish language tasks.
  • Domain adaptation for specific Spanish corpora.
  • Research into language-specific model pretraining and adaptation.

Finetunes
1 models