TucanoBR/Tucano-1b1

Tucano-1b1: A Portuguese-Native Language Model

Tucano-1b1 is a 1.1 billion parameter decoder-transformer model developed by TucanoBR, specifically pretrained for the Portuguese language. It is part of the Tucano series, which focuses on native Portuguese language modeling. The model was trained on the extensive GigaVerbo dataset, a 200 billion token collection of deduplicated Portuguese text corpora, and features a context length of 2048 tokens.

Key Capabilities

• Native Portuguese Pretraining: Designed from the ground up for Portuguese, leveraging a massive, dedicated dataset.
• Research Foundation: Intended primarily as a base for research and development in Portuguese NLP, allowing for controlled experiments on pretraining effects.
• Fine-tuning Potential: Can be fine-tuned and adapted for deployment under the Apache 2.0 license, though users are advised to conduct their own risk and bias assessments.

Good for

• Portuguese NLP Research: Ideal for academics and researchers exploring language modeling specific to Portuguese.
• Comparative Experiments: Provides a stable checkpoint for evaluating the impact of active pretraining on various benchmarks.
• Developing Portuguese-centric Applications: Suitable as a foundational model for building custom applications that require strong Portuguese language understanding and generation, after appropriate fine-tuning and evaluation.

Limitations

It is important to note that Tucano-1b1 is not intended for direct deployment as an out-of-the-box product. It is designed for Portuguese only and is not suitable for other languages. Like many large language models, it is susceptible to hallucinations, biases, and can produce unreliable code or repetitive/verbose responses. Users are urged to perform thorough risk analysis for real-world applications.