Overview

Tensoic/Mistral-7B-v0.1-alpaca-2k-test is an experimental fine-tune of the Mistral 7B base model, created by Tensoic. This model represents an initial attempt at adapting Mistral 7B for instruction-following tasks using the Alpaca-2k-test dataset. It serves as a foundational model for developers to explore the capabilities of Mistral 7B when fine-tuned with a specific instruction dataset.

Training Details

The model was trained with a learning rate of 0.0002 over 3 epochs, utilizing a total batch size of 16 across 8 GPUs. Key hyperparameters included a cosine learning rate scheduler with 10 warmup steps and an Adam optimizer. The training leveraged Transformers 4.34.0.dev0, Pytorch 2.0.1+cu117, Datasets 2.14.5, and Tokenizers 0.14.0.

Good For

• Experimentation: Ideal for developers looking to test and understand the behavior of Mistral 7B fine-tuned on an Alpaca-style dataset.
• Instruction-Following Tasks: Suitable for initial exploration of instruction-based prompts, given its fine-tuning on the Alpaca-2k-test dataset.
• Benchmarking: Can be used as a baseline for comparing against other instruction-tuned Mistral 7B variants.