manucif/Llama-3.2-1B
The manucif/Llama-3.2-1B model, developed by Meta, is a 1.23 billion parameter multilingual large language model built on an optimized transformer architecture. It is instruction-tuned for multilingual dialogue use cases, excelling in agentic retrieval and summarization tasks. This model supports a 32768 token context length and is optimized for deployment in constrained environments like mobile devices.
Loading preview...
Overview
manucif/Llama-3.2-1B is a 1.23 billion parameter multilingual large language model from Meta's Llama 3.2 collection. It utilizes an optimized transformer architecture and is instruction-tuned using supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) to align with human preferences for helpfulness and safety. The model was pretrained on up to 9 trillion tokens of publicly available online data, with a knowledge cutoff of December 2023. It features Grouped-Query Attention (GQA) for improved inference scalability and supports a context length of 32768 tokens.
Key Capabilities
- Multilingual Dialogue: Optimized for assistant-like chat and agentic applications in multiple languages, including English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai.
- Retrieval and Summarization: Excels in tasks such as knowledge retrieval and summarization.
- Quantization: Includes various quantization schemes like 4-bit groupwise for weights and 8-bit dynamic for activations, along with Quantization-Aware Training (QAT) and SpinQuant, significantly reducing model size and improving inference speed on ARM CPUs.
- Performance: Demonstrates strong performance on benchmarks like MMLU, GSM8K, and MATH, even in its smaller 1B size, and shows substantial improvements in decode and prefill speeds with quantization.
Good For
- Commercial and Research Use: Intended for a wide range of commercial and research applications.
- Constrained Environments: Particularly suited for deployment in environments with limited compute resources, such as mobile devices, due to its small size and optimized quantization methods.
- Agentic Applications: Ideal for building agentic systems requiring multilingual chat, knowledge retrieval, and summarization capabilities.