wvnvwn/Meta-Llama-3-8B-Instruct-fedavg-v0
wvnvwn/Meta-Llama-3-8B-Instruct-fedavg-v0 is an 8 billion parameter instruction-tuned causal language model, based on Meta-Llama-3-8B-Instruct. Developed by wvnvwn, this model was created through federated LoRA fine-tuning and adapter aggregation using the fedavg algorithm. It is designed for reproducible evaluation, offering a merged full-weight model rather than an adapter-only checkpoint, and is suitable for tasks requiring a robust instruction-following LLM.
Loading preview...
Model Overview
wvnvwn/Meta-Llama-3-8B-Instruct-fedavg-v0 is an 8 billion parameter instruction-tuned model derived from the meta-llama/Meta-Llama-3-8B-Instruct base model. This model is notable for its training procedure, which involved federated LoRA fine-tuning followed by adapter aggregation using the fedavg algorithm. The resulting PEFT LoRA adapter was then merged into the base model, providing a full-weight model for direct and reproducible evaluation without needing separate adapter loading.
Key Training Details
- Procedure: Federated LoRA fine-tuning and adapter aggregation.
- Algorithm:
fedavg - Training Data:
data_hetero_with_4_tasks - Clients: 8
- Communication Rounds: 3
- Local Epochs: 3
- Local Batch Size: 256
- LoRA Configuration: Rank 16, Alpha 16, targeting
up_proj,v_proj,gate_proj,q_proj,k_proj,o_proj,down_projmodules.
Usage and Evaluation
This model is provided as a merged full-weight checkpoint, simplifying deployment and evaluation. It can be loaded using the Hugging Face transformers library with AutoModelForCausalLM and AutoTokenizer. A bundled FSL evaluation wrapper is available for reproducible performance assessment. Users should adhere to the base model's license and access policies for redistribution and usage.