prism-ml/Bonsai-27B-unpacked

Bonsai-27B-unpacked: FP16 Safetensors

This repository provides the prism-ml/Bonsai-27B-unpacked model, an FP16 safetensors version of the 1-bit Bonsai 27B architecture. Developed by PrismML, this 27 billion parameter model is designed for compatibility with standard HuggingFace tooling and frameworks that do not natively support 1-bit weights.

Key Characteristics

• Format: FP16 safetensors, compatible with stock HuggingFace environments.
• Size: Full-size (~54 GB) due to FP16 precision, unlike the highly compressed 1-bit native versions.
• Context Length: Supports a 32768 token context window.

Important Considerations

It is crucial to note that this unpacked FP16 version does not offer the memory reduction or interactive decoding speeds of the native 1-bit Bonsai models. The primary benefits of Bonsai, such as a 14.2x memory reduction to 3.9 GB and interactive decoding on everyday laptops (e.g., 44 tok/s on an M5 Pro), are exclusive to the optimized 1-bit formats. This unpacked version is intended as a bridge for users whose frameworks lack native 1-bit support.

Recommended Alternatives (for optimal performance)

For users seeking the full advantages of the Bonsai architecture, PrismML strongly recommends using the native 1-bit models:

• Bonsai-27B-mlx-1bit: Optimized 1-bit MLX for Apple Silicon (Mac, iPhone, iPad).
• 1-bit GGUF (Q1_0_g128): For llama.cpp (CUDA, Metal, CPU).

Additionally, a quality-oriented ternary variant is available:

• Ternary-Bonsai-27B-mlx-2bit: A 2-bit version offering ~95% of FP16 quality at ~7.2 GB, suitable for laptops and GPUs.