Model Overview

Locutusque/lr-experiment1-7B is a 7 billion parameter language model built upon the Mistral architecture, developed by Locutusque. This model is the first in a research series aimed at identifying the most effective learning rates for fine-tuning Mistral-based models. It was fine-tuned using QLoRA for 3 epochs on the CollectiveCognition/chats-data-2023-09-22 dataset, specifically employing a learning rate of 2e-5 with a cosine scheduler and no warmup steps.

Key Characteristics

• Base Model: Locutusque/Hercules-2.0-Mistral-7B
• Fine-tuning Method: QLoRA for 3 epochs
• Learning Rate: 2e-5 with a cosine scheduler
• Context Length: 8192 tokens
• Purpose: Serves as an experimental benchmark to evaluate the impact of specific learning rate configurations on model performance.

Evaluation Highlights

Initial evaluations on the agieval_nous benchmark show an acc score of 0.3645 and an acc_norm score of 0.3468. Specific sub-tasks within agieval_nous include:

• agieval_aqua_rat: acc 0.2283
• agieval_logiqa_en: acc 0.2965
• agieval_lsat_lr: acc 0.4039
• agieval_sat_en: acc 0.6408

These results provide a baseline for comparison with future models in the lr-experiment series, which will explore different learning rate strategies.