RthItalia/NanoLLM-Qwen2.5-3B-v3.1

TEXT GENERATIONPricing:Input $0.32 / Cached $0.064 / Output $1.6Concurrent Unit Cost:1Model Size:3.1BQuant:BF16Context Size:32kTool Calling:SupportedPublished:Apr 6, 2026License:otherArchitecture:Transformer Featherless Exclusive Cold

RthItalia/NanoLLM-Qwen2.5-3B-v3.1 is a compact, self-contained variant of the Qwen2.5-3B-Instruct model, developed by RthItalia. This 3 billion parameter model is optimized for reduced VRAM usage and disk size, achieving a peak generation VRAM of ~2.44 GB compared to the 8-bit baseline's ~3.21 GB. It employs a mixed runtime policy with QKV projections in FP16 and other components in a Nano compact format, making it suitable for memory-constrained environments while preserving quality.

Loading preview...

NanoLLM-Qwen2.5-3B-v3.1: Compact and Efficient

This model, developed by RthItalia, is a specialized, compact variant derived from Qwen/Qwen2.5-3B-Instruct. It is designed to provide a significant reduction in VRAM usage and disk size without a substantial compromise in quality, making it ideal for resource-constrained deployments.

Key Optimizations and Features

  • Mixed Runtime Policy: Utilizes fp16 for q_proj, k_proj, v_proj and a Nano compact format for o_proj and other transformer components. This strategy balances compression with performance.
  • VRAM Efficiency: Achieves a peak generation VRAM of approximately 2.44 GB, a notable improvement over the 8bit baseline's ~3.21 GB.
  • Disk Size Reduction: The model size is 2.3432 GB.
  • Custom Head Implementation: Features a NanoTiedHead that ties output logits to the quantized embedding table, further optimizing memory.
  • Self-Contained: This release is an exported, self-contained folder that loads directly with the transformers library.

Quality and Limitations

The model's quality was validated against a true 8bit baseline using a small internal prompt suite. While it restored acceptable behavior on this reference set, it is not claimed to match the full original model in all tasks or outperform the base model. It should be considered a compact experimental runtime artifact, specifically tuned around Qwen/Qwen2.5-3B-Instruct, rather than a broad architectural proof.

Good for

  • Deployments requiring reduced VRAM and disk footprint.
  • Use cases where memory efficiency is critical, such as edge devices or environments with limited GPU resources.
  • Developers seeking a compact, validated variant of Qwen2.5-3B-Instruct for specific applications.

Finetunes
1 models