Model Overview

RikkiXu/zephyr-7b-sft-full is a 7 billion parameter language model derived from the mistralai/Mistral-7B-v0.1 base model. It has undergone supervised fine-tuning (SFT) using a specific generator dataset, which aims to refine its ability to produce high-quality text.

Training Details

The model was trained with a learning rate of 2e-05, a train_batch_size of 16, and an eval_batch_size of 8, utilizing 8 devices for a total effective batch size of 128. The training process involved 1 epoch, using an Adam optimizer with cosine learning rate scheduling and a warmup ratio of 0.1. During training, a validation loss of 0.9406 was achieved.

Framework Versions

Key frameworks used include Transformers 4.41.1, Pytorch 2.1.2+cu118, Datasets 2.16.1, and Tokenizers 0.19.1.

Intended Uses

This model is primarily intended for applications that benefit from a fine-tuned Mistral 7B variant, particularly in scenarios where enhanced text generation performance is desired due to its SFT on a generator dataset.