uukuguy/Mistral-7B-OpenOrca-lora-merged
The uukuguy/Mistral-7B-OpenOrca-lora-merged model is a regenerated 7 billion parameter language model combining the Mistral-7B-v0.1 base with a LoRA adapter extracted from the Mistral-7B-OpenOrca model. This model is designed to verify if a merged LoRA can achieve comparable performance to the original fine-tuned model. It aims to serve as a component in a toolkit for dynamically loading and switching multiple LoRA modules based on user queries.
Loading preview...
Overview
This model, uukuguy/Mistral-7B-OpenOrca-lora-merged, is a 7 billion parameter language model created by merging the base model Mistral-7B-v0.1 with a LoRA (Low-Rank Adaptation) module. The LoRA module was specifically extracted from the Mistral-7B-OpenOrca model, which is known for its efficient parameter fine-tuning. The primary goal behind this merged model is to validate whether a LoRA-merged model can replicate the performance of its original fine-tuned counterpart.
Key Capabilities & Purpose
- LoRA Verification: The model's core purpose is to test the efficacy of LoRA extraction and merging, aiming to achieve performance comparable to the original
Mistral-7B-OpenOrcamodel. - Foundation for Multi-LoRA Systems: It serves as a foundational step towards developing a toolkit capable of loading and dynamically switching multiple LoRA modules based on user queries, optimizing response generation.
Performance Insights
Evaluations show that the r=256 configuration of the LoRA-merged model achieves competitive scores, sometimes surpassing the original Mistral-7B-OpenOrca in specific benchmarks:
- MMLU_acc (5-shot):
64.28(vs.61.42for original) - HellaSwag_acc_norm (10-shot):
84(vs.83for original) - Open LLM Score:
65.81(vs.65.11for original)
Training Details
The model was trained using bitsandbytes quantization with load_in_4bit: True, bnb_4bit_quant_type: nf4, and bnb_4bit_use_double_quant: True, leveraging PEFT 0.5.0.