Treese/RQwen3-751M-Base

TEXT GENERATIONPricing:Input $0.04 / Cached $0.008 / Output $0.08Concurrent Unit Cost:1Model Size:0.8BQuant:BF16Context Size:32kTool Calling:SupportedPublished:Sep 15, 2026License:apache-2.0Architecture:Transformer Open Weights Featherless Exclusive Cold

Treese/RQwen3-751M-Base is a 751-million-parameter language model, architecturally matching Qwen3-0.6B, pretrained from scratch on approximately 13 billion tokens of curated educational data. This base model, developed by Treese, is a next-token predictor with a 2,048-token context length, designed for research and educational purposes rather than instruction-following or chat. It shares no weights with existing Qwen models, focusing on demonstrating learned commonsense-continuation ability.

Loading preview...

RQwen3-751M-Base Overview

RQwen3-751M-Base is a 751-million-parameter language model developed by Treese, pretrained from scratch on approximately 13 billion tokens of curated educational data. It adopts the Qwen3-0.6B architecture (RoPE, GQA, SwiGLU, RMSNorm, QK-Norm) but shares no weights with any released Qwen model, utilizing the same architecture and tokenizer. This is a base model designed as a next-token predictor, not an instruction-tuned assistant, and is intended primarily for research and educational artifacts.

Key Capabilities

  • Architectural Fidelity: Matches Qwen3-0.6B's architecture, ensuring compatibility with existing Qwen3 tooling (e.g., vLLM, llama.cpp converters).
  • Pretrained from Scratch: Offers a unique opportunity to study model behavior and scaling laws from a clean slate, without inherited biases or knowledge from other models.
  • Commonsense Continuation: Demonstrates learned commonsense-continuation ability, scoring +6.5 points over chance on HellaSwag, indicating effective learning from its educational dataset.
  • Standard transformers Integration: Loads seamlessly via transformers' Qwen3ForCausalLM class without needing trust_remote_code.

Good For

  • Research and Education: Ideal for studying language model training, architecture, and the impact of data curation on base model capabilities.
  • Experimentation with Base Models: Provides a foundation for further fine-tuning, instruction-tuning, or alignment experiments.
  • Understanding Scaling Laws: Useful for exploring how models of this size and training data volume perform on various tasks, particularly in comparison to larger or differently trained models.

Limitations

  • Not an Assistant: Lacks instruction-tuning, SFT, or RLHF; it does not reliably follow instructions or answer questions.
  • Fact Fabrication: Prone to generating authoritative-sounding but incorrect statements due to its training scale.
  • Short Context: Trained with a 2,048-token context length; performance beyond this is untested and likely poor.
  • English-only in Practice: While the tokenizer is multilingual, the training data is predominantly English.
  • Unfiltered Data: Contains biases and potential toxicity from web-scale corpora, as no safety tuning has been applied.

Finetunes
1 models