TucanoBR/Tucano-1b1-Instruct
Tucano-1b1-Instruct is a 1.1 billion parameter instruction-tuned decoder-transformer model developed by TucanoBR, natively pretrained in Portuguese. It was fine-tuned using Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) on various instruction datasets. This model is designed for research and development in Portuguese language modeling, serving as a foundation for comparative experiments and further fine-tuning.
Loading preview...
Tucano-1b1-Instruct Overview
Tucano-1b1-Instruct is a 1.1 billion parameter instruction-tuned decoder-transformer model, part of the Tucano series, which is natively pretrained in Portuguese. The base model, Tucano-1b1, was trained on GigaVerbo, a 200 billion token Portuguese text corpus. This instruction-tuned version underwent a two-stage fine-tuning process:
Key Capabilities & Training
- Native Portuguese Pretraining: Built upon a foundation model pretrained specifically on a large Portuguese corpus (GigaVerbo).
- Instruction Tuning: Fine-tuned using Supervised Fine-Tuning (SFT) with a concatenation of three instruction datasets, followed by Direct Preference Optimization (DPO).
- Research Foundation: Intended primarily for research and development in Portuguese language modeling, offering a controlled setting for comparative experiments.
- Context Length: Supports a context length of 2048 tokens.
Intended Use Cases
- Research and Development: Ideal for exploring and advancing neural text generation in Portuguese.
- Comparative Experiments: Useful for studying the effects of active pretraining on benchmark performance.
- Foundation for Fine-tuning: Can be adapted and further fine-tuned for specific deployment scenarios, provided the use adheres to the Apache 2.0 license and includes independent risk and bias assessments.
Limitations
It is important to note that Tucano-1b1-Instruct is not intended for out-of-the-box deployment in human-facing interactions. It is designed exclusively for the Portuguese language and has not been fine-tuned for general downstream tasks. Like many large language models, it is subject to hallucinations, biases, toxicity, and may produce unreliable code or exhibit repetition.