Locutusque/lr-experiment1-7B
Locutusque/lr-experiment1-7B is a 7 billion parameter Mistral-based language model developed by Locutusque, fine-tuned with QLoRA for 3 epochs on conversational data. This model is part of a research series to determine optimal learning rates for Mistral fine-tuning, specifically using a 2e-5 learning rate with a cosine scheduler. It is designed for general conversational tasks and serves as a benchmark for ongoing learning rate experiments.
Loading preview...
Model Overview
Locutusque/lr-experiment1-7B is a 7 billion parameter language model built upon the Mistral architecture, developed by Locutusque. This model is the first in a research series aimed at identifying the most effective learning rates for fine-tuning Mistral-based models. It was fine-tuned using QLoRA for 3 epochs on the CollectiveCognition/chats-data-2023-09-22 dataset, specifically employing a learning rate of 2e-5 with a cosine scheduler and no warmup steps.
Key Characteristics
- Base Model: Locutusque/Hercules-2.0-Mistral-7B
- Fine-tuning Method: QLoRA for 3 epochs
- Learning Rate: 2e-5 with a cosine scheduler
- Context Length: 8192 tokens
- Purpose: Serves as an experimental benchmark to evaluate the impact of specific learning rate configurations on model performance.
Evaluation Highlights
Initial evaluations on the agieval_nous benchmark show an acc score of 0.3645 and an acc_norm score of 0.3468. Specific sub-tasks within agieval_nous include:
agieval_aqua_rat:acc0.2283agieval_logiqa_en:acc0.2965agieval_lsat_lr:acc0.4039agieval_sat_en:acc0.6408
These results provide a baseline for comparison with future models in the lr-experiment series, which will explore different learning rate strategies.
Top 3 parameter combinations used by Featherless users for this model. Click a tab to see each config.