Overview
This model, uukuguy/Mistral-7B-OpenOrca-lora-merged, is a 7 billion parameter language model created by merging the base model Mistral-7B-v0.1 with a LoRA (Low-Rank Adaptation) module. The LoRA module was specifically extracted from the Mistral-7B-OpenOrca model, which is known for its efficient parameter fine-tuning. The primary goal behind this merged model is to validate whether a LoRA-merged model can replicate the performance of its original fine-tuned counterpart.
Key Capabilities & Purpose
- LoRA Verification: The model's core purpose is to test the efficacy of LoRA extraction and merging, aiming to achieve performance comparable to the original
Mistral-7B-OpenOrca model. - Foundation for Multi-LoRA Systems: It serves as a foundational step towards developing a toolkit capable of loading and dynamically switching multiple LoRA modules based on user queries, optimizing response generation.
Performance Insights
Evaluations show that the r=256 configuration of the LoRA-merged model achieves competitive scores, sometimes surpassing the original Mistral-7B-OpenOrca in specific benchmarks:
- MMLU_acc (5-shot):
64.28 (vs. 61.42 for original) - HellaSwag_acc_norm (10-shot):
84 (vs. 83 for original) - Open LLM Score:
65.81 (vs. 65.11 for original)
Training Details
The model was trained using bitsandbytes quantization with load_in_4bit: True, bnb_4bit_quant_type: nf4, and bnb_4bit_use_double_quant: True, leveraging PEFT 0.5.0.