ignos/Mistral-T5-7B-v1: Instruction-Tuned Mistral Model

This model, developed by Ignos, is a 7 billion parameter language model built upon the Mistral architecture. It is a fine-tuned version of the Toten5/Marcoroni-neural-chat-7B-v2 base model, specifically optimized for improving instructional behavior.

Key Capabilities

• Instruction Following: Designed to enhance responses to instructional prompts.
• Mistral Architecture: Benefits from the efficient and capable Mistral model design.
• QLoRA Fine-tuning: Utilizes the QLoRA approach for efficient adaptation and merging with the base model.
• Apache-2.0 License: Offers flexible usage under an open-source license.

Training Details

The model was trained using the QLoRA method, leveraging the tatsu-lab/alpaca dataset for instruction-tuning. Training was conducted on a robust compute infrastructure featuring 3 x RTX 4090 GPUs, 48 vCPUs, and 377 GB RAM, utilizing Axolotl 0.3.0 and PEFT 0.6.0 frameworks. The model aims to provide improved performance in instructional contexts, building on its 8192 token context length.