True2456/Qwen3.8-27B-ANE

VISIONPricing:Input $1.6 / Cached $0.15 / Output $12Concurrent Unit Cost:2Model Size:27BQuant:FP8Context Size:32kTool Calling:SupportedPublished:Sep 14, 2026License:apache-2.0Architecture:Transformer0.0K Open Weights Featherless Exclusive Cold

True2456/Qwen3.8-27B-ANE is an INT4 quantized version of the 27 billion parameter Qwen3.8 model, specifically optimized by True2456 for efficient inference on Apple Silicon's Neural Engine. This package runs learned matrix multiplications, Gated DeltaNet recurrence, attention, and MLP tails entirely on the Apple Neural Engine, achieving ultra-low power consumption and a stable 21.0 GB memory footprint. It is designed for local deployment on macOS devices, offering a performant solution for agent turns and chat applications without utilizing the Metal GPU.

Loading preview...

Overview

True2456/Qwen3.8-27B-ANE is a specialized INT4 runtime package for the 27 billion parameter Qwen3.8 model, engineered by True2456 for exclusive execution on Apple Silicon's Neural Engine (ANE). This model leverages the AppleNeuralEngine.framework to offload core computational tasks, including learned matrix multiplications, Gated DeltaNet recurrence, attention, and MLP tails, ensuring zero Metal GPU utilization. The package is not a standard Transformers-loadable full-weight dump but rather a pre-quantized INT4 version, with host tensors in model.safetensors and 27B INT4 layer weights in quant_cache/.

Key Capabilities & Performance

  • Apple Neural Engine Exclusive: Achieves 100% on-chip ANE inference, completely bypassing the Metal GPU.
  • Ultra-Low Power Consumption: Sustained ANE power draw of approximately 2.2–2.9 W, with total package power under 10 W.
  • Stable Memory Footprint: Maintains a consistent physical memory footprint of 21.0 GB across varying context lengths (1k to 8k+ tokens).
  • Efficient Local Deployment: Designed for local chat and serving on macOS, with support for optional speculative decoding using mtp.safetensors.
  • Architecture: Features 64 layers in a 3:1 pattern (48 Gated DeltaNet and 16 full-attention layers), with projections, convolutions, and SwiGLU MLPs compiled as MIL graphs for the ANE.

Use Cases

  • On-device AI Applications: Ideal for running large language models directly on Apple Silicon devices with minimal power draw.
  • Agent Turns & Chatbots: Provides efficient token generation rates (3.4–4.3 tok/s for greedy decode) suitable for interactive applications.
  • Edge Computing: Enables powerful LLM capabilities in environments where GPU resources are limited or power efficiency is critical.

Finetunes
1 models