mlfoundations-dev/oh_v1.3_alpaca_x4
The mlfoundations-dev/oh_v1.3_alpaca_x4 model is an 8 billion parameter language model fine-tuned from Meta-Llama-3.1-8B. This model was fine-tuned on the mlfoundations-dev/oh_v1.3_alpaca_x4 dataset, achieving a validation loss of 0.7339. It is designed for general language generation tasks, leveraging the Llama 3.1 architecture.
Loading preview...
Overview
The mlfoundations-dev/oh_v1.3_alpaca_x4 is an 8 billion parameter language model derived from the Meta-Llama-3.1-8B architecture. It has been fine-tuned using the mlfoundations-dev/oh_v1.3_alpaca_x4 dataset, demonstrating a final validation loss of 0.7339.
Training Details
The model underwent training with the following key hyperparameters:
- Learning Rate: 5e-06
- Batch Size: 8 (train and eval), with a total effective training batch size of 512 due to gradient accumulation.
- Optimizer: AdamW with betas=(0.9, 0.999) and epsilon=1e-08.
- Epochs: 3.0
- Devices: Trained across 8 GPUs.
Training results show a progressive decrease in loss over epochs, with the validation loss stabilizing around 0.7339 by the final epoch.
Intended Uses
This model is suitable for general natural language processing tasks where a fine-tuned Llama 3.1 base model is beneficial. Specific use cases would depend on the characteristics of the mlfoundations-dev/oh_v1.3_alpaca_x4 dataset, which is not detailed in the provided information.