Model Overview

ParetoQaft/1B-base is a 1.23 billion parameter model from Meta's Llama 3.2 collection, designed for multilingual text generation. It utilizes an optimized transformer architecture and boasts a substantial context length of 32,768 tokens. The model was pretrained on up to 9 trillion tokens of diverse public online data, incorporating knowledge distillation from larger Llama 3.1 models to enhance performance.

Key Capabilities

• Multilingual Text Generation: Supports English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai, with potential for fine-tuning in other languages.
• Optimized Architecture: Features Grouped-Query Attention (GQA) for improved inference scalability.
• Foundation Model: Intended for commercial and research use, adaptable for various natural language generation tasks.
• Quantization Support: Designed with quantization schemes (4-bit groupwise for weights, 8-bit dynamic for activations) for efficient deployment in constrained environments, including mobile devices.

Good For

• Research and Commercial Applications: A versatile base model for a wide range of NLP tasks.
• Multilingual Development: Ideal for applications requiring understanding and generation across multiple languages.
• Resource-Constrained Environments: The 1B size and quantization options make it suitable for on-device deployment with limited compute resources.