wvnvwn/Mistral-7B-Instruct-v0.3-fedavg-v0
The wvnvwn/Mistral-7B-Instruct-v0.3-fedavg-v0 is a 7 billion parameter instruction-tuned causal language model based on the Mistral-7B-Instruct-v0.3 architecture. This model was created by wvnvwn through federated LoRA fine-tuning and adapter aggregation, resulting in a merged full-weight model. It is designed for reproducible evaluation and general instruction-following tasks, leveraging a 4096 token context length.
Loading preview...
Model Overview
The wvnvwn/Mistral-7B-Instruct-v0.3-fedavg-v0 is a 7 billion parameter instruction-tuned causal language model. It is derived from the mistralai/Mistral-7B-Instruct-v0.3 base model, enhanced through a federated learning approach.
Key Characteristics
- Federated Fine-tuning: The model was fine-tuned using a federated LoRA (Low-Rank Adaptation) procedure, specifically employing the
fedavgalgorithm. - Merged Adapter: A PEFT LoRA adapter was applied and then merged into the base model, creating a full-weight model for simplified and reproducible deployment without requiring separate adapter loading.
- Training Data: Training involved a heterogeneous dataset (
data_hetero_with_4_tasks) distributed across 8 clients over 3 communication rounds. - Reproducible Evaluation: The merged nature of the model facilitates straightforward and consistent evaluation.
Usage
This model is suitable for general instruction-following tasks, leveraging its 7 billion parameters and 4096 token context window. It can be loaded and utilized with the Hugging Face Transformers library, supporting torch.bfloat16 for efficient inference.