ParetoQaft/1B-base
The ParetoQaft/1B-base model is a 1.23 billion parameter Llama 3.2 collection multilingual large language model developed by Meta, featuring an optimized transformer architecture and a 32,768 token context length. It is pretrained on up to 9 trillion tokens of publicly available online data, with knowledge distillation from larger Llama 3.1 models. This base model is designed for commercial and research use in multilingual text generation, serving as a foundation for various natural language generation tasks.
Loading preview...
Model Overview
ParetoQaft/1B-base is a 1.23 billion parameter model from Meta's Llama 3.2 collection, designed for multilingual text generation. It utilizes an optimized transformer architecture and boasts a substantial context length of 32,768 tokens. The model was pretrained on up to 9 trillion tokens of diverse public online data, incorporating knowledge distillation from larger Llama 3.1 models to enhance performance.
Key Capabilities
- Multilingual Text Generation: Supports English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai, with potential for fine-tuning in other languages.
- Optimized Architecture: Features Grouped-Query Attention (GQA) for improved inference scalability.
- Foundation Model: Intended for commercial and research use, adaptable for various natural language generation tasks.
- Quantization Support: Designed with quantization schemes (4-bit groupwise for weights, 8-bit dynamic for activations) for efficient deployment in constrained environments, including mobile devices.
Good For
- Research and Commercial Applications: A versatile base model for a wide range of NLP tasks.
- Multilingual Development: Ideal for applications requiring understanding and generation across multiple languages.
- Resource-Constrained Environments: The 1B size and quantization options make it suitable for on-device deployment with limited compute resources.