TucanoBR/Tucano-1b1-Instruct

Model Overview

Tucano-1b1-Instruct is a 1.1 billion parameter decoder-transformer model, part of the Tucano series, natively pretrained in Portuguese by TucanoBR. It was initially trained on GigaVerbo, a 200 billion token Portuguese text corpus. The instruction-tuned version underwent a two-stage fine-tuning process: Supervised Fine-Tuning (SFT) using a concatenation of three instruction datasets, followed by Direct Preference Optimization (DPO).

Key Capabilities

• Native Portuguese Language Modeling: Specifically designed and pretrained for the Portuguese language.
• Instruction Following: Fine-tuned with SFT and DPO to understand and respond to instructions.
• Research Foundation: Intended as a base for research and development in Portuguese NLP, allowing for comparative experiments.

Intended Uses

• Research and Development: Ideal for foundational research in Portuguese language modeling.
• Comparative Experiments: Provides a controlled setting for evaluating active pretraining effects on benchmarks.
• Fine-tuning Base: Can be adapted and fine-tuned for specific deployments under the Apache 2.0 license, with users responsible for risk and bias assessments.

Limitations

• Portuguese Only: Unsuitable for text generation in other languages.
• Not for Direct Deployment: Not an out-of-the-box product for human-facing interactions.
• Potential for Hallucinations and Bias: Inherits biases from training data and can produce misleading or toxic content.
• Unreliable Code Generation: May produce incorrect code snippets.
• Repetition and Verbosity: Can exhibit repetitive or verbose responses.