kirilldual0987/MIXdevAI-llama-quantized
kirilldual0987/MIXdevAI-llama-quantized is a 1 billion parameter Llama 3-based language model, created by Kolyadual, from the Newton bot family. This model is a SLERP merge of meta-llama/Llama-3.2-1B-Instruct and KingNish/Reasoning-Llama-1b-v0.1, optimized for efficient local deployment across various hardware. It provides a comprehensive set of GGUF quantizations, making it suitable for diverse applications requiring a balance of performance and resource usage.
Loading preview...
Overview
kirilldual0987/MIXdevAI-llama-quantized provides GGUF-quantized versions of the original Kolyadual/MIXdevAI-llama model. This is a lightweight 1 billion parameter AI model based on Llama 3 from the Newton bot family, developed by Kolyadual. It is a merge of two architectures, meta-llama/Llama-3.2-1B-Instruct and KingNish/Reasoning-Llama-1b-v0.1, using the SLERP method with bfloat16 dtype.
Key Features
- Quantized for Efficiency: All files are converted and quantized using
llama.cppfor fast local execution. - Broad Device Compatibility: Designed for deployment on various devices, including CPU, Apple Silicon (Metal), and GPU, via platforms like Ollama, LM Studio, and
llama.cpp. - Diverse Quantization Formats: Offers a wide range of GGUF quantization formats (e.g., F16, Q8_0, Q6_K, Q4_K_M, IQ-series) to balance quality, speed, and memory requirements.
Recommended Usage
- Q4_K_M: This format is highlighted as the recommended "golden mean" for most tasks and devices, offering a good balance between compression and model intelligence.
- IQ-series: Innovative formats like IQ3_XS and IQ4_NL provide excellent quality at extremely small sizes, ideal for resource-constrained environments like smartphones.
For detailed instructions on using the base model via Transformers and the original YAML merge configuration, refer to the original repository by Kolyadual.