Model Overview

The thangvip/pirate-0.5 is a 0.6 billion parameter language model, derived from a fine-tuning process applied to the sail/Sailor-0.5B base model. It supports a substantial context length of 32,768 tokens, making it capable of processing longer sequences of text.

Key Characteristics

• Base Model: Fine-tuned from sail/Sailor-0.5B.
• Parameter Count: 0.6 billion parameters.
• Context Length: 32,768 tokens.
• Training Details: The model was trained with a learning rate of 0.0001, a batch size of 1 (accumulated to 16), and utilized the Adam optimizer over 1 epoch. It was developed using Transformers 4.39.0 and PyTorch 2.2.1.

Current Limitations

Detailed information regarding the specific dataset used for fine-tuning, its intended uses, and known limitations is not provided in the available documentation. Users should exercise caution and conduct thorough testing for specific applications.