Model Overview

The rishiraj/meow model is a 10.7 billion parameter language model, fine-tuned from the upstage/SOLAR-10.7B-Instruct-v1.0 base model. It was specifically trained on the HuggingFaceH4/no_robots dataset.

Training Details

The model underwent a single epoch of training with a learning rate of 2e-05, a train_batch_size of 4, and a gradient_accumulation_steps of 128, resulting in a total_train_batch_size of 512. The optimizer used was Adam with default betas and epsilon, and a cosine learning rate scheduler. During training, a validation loss of 2.3831 was observed.

Framework Versions

Training utilized:

• Transformers 4.37.0.dev0
• Pytorch 2.1.1+cu121
• Datasets 2.14.6
• Tokenizers 0.15.0
• PEFT 0.6.1

Intended Use Cases

Given its fine-tuning on an instruction dataset, this model is suitable for general instruction-following tasks. However, specific intended uses and limitations are not detailed in the provided information.