prism-ml/Ternary-Bonsai-27B-unpacked
The prism-ml/Ternary-Bonsai-27B-unpacked is an FP16 safetensors version of the 27 billion parameter Ternary Bonsai model, developed by prism-ml. This unpacked version is provided for compatibility with standard HuggingFace tooling and frameworks that do not yet support the native packed ternary format. It offers a 32768 token context length but does not retain the memory or performance benefits of the recommended packed ternary models.
Loading preview...
Ternary Bonsai 27B Unpacked FP16
This repository provides the prism-ml/Ternary-Bonsai-27B-unpacked model as FP16 safetensors, designed for compatibility with standard HuggingFace environments. It represents the full 27 billion parameter model with a 32768 token context length.
Key Characteristics
- Format: FP16 safetensors, making it compatible with generic HuggingFace tooling.
- Purpose: Serves as an interim solution for users whose frameworks do not yet support the specialized packed ternary format of the Ternary Bonsai models.
- Limitations: Unlike the natively packed versions, this unpacked FP16 model does not offer the significant memory footprint reduction (e.g., 7.2 GB vs. 54 GB) or the interactive decoding performance benefits (e.g., 26 tok/s on M5 Pro) that are central to the Ternary Bonsai architecture.
When to Use
This unpacked version is specifically for developers who:
- Require a standard FP16 HuggingFace format for immediate integration.
- Are working with frameworks that lack support for the optimized 2-bit hybrid-attention kernels found in the recommended packed models.
Note: For optimal performance, reduced memory footprint, and faster inference, it is strongly recommended to use the natively packed Ternary Bonsai models, such as Ternary-Bonsai-27B-mlx-2bit for Apple Silicon or Ternary GGUF (Q2_0_g128) for llama.cpp.