Model Overview

mrm8488/limstral-7B-v0.1 is a 7 billion parameter language model built upon the Mistral-7B-v0.1 architecture. It has been specifically fine-tuned for instruction following capabilities, leveraging the LIMA dataset.

Key Capabilities

• Instruction Following: Excels at generating responses that adhere to explicit instructions provided in the prompt.
• Causal Language Modeling: Inherits the strong generative abilities of its Mistral base model.

Training Details

The model was fine-tuned using LoRA PEFT technique on 8-bit loaded weights, utilizing huggingface/peft and trl/sft libraries. The training involved 2 epochs on a single A100 (40GB) GPU, with a max_seq_length of 2048 tokens. Key hyperparameters included a learning rate of 0.0002 and a total batch size of 128.

Good For

• Instruction-based tasks: Ideal for applications requiring the model to follow specific commands or answer questions based on given instructions.
• Conversational AI: Can be used as a foundation for chatbots or virtual assistants where precise instruction adherence is crucial.
• Research and Development: Provides a fine-tuned Mistral-7B variant for exploring instruction-following performance with the LIMA dataset.