mlfoundations-dev/oh_v1.3_alpaca_x2

Model Overview

The mlfoundations-dev/oh_v1.3_alpaca_x2 is an 8 billion parameter language model, fine-tuned from the robust Meta-Llama-3.1-8B base architecture. This model underwent specific training on the mlfoundations-dev/oh_v1.3_alpaca_x2 dataset, demonstrating a final validation loss of 0.7331 over 3 epochs.

Training Details

Training was conducted using a multi-GPU setup (8 devices) with a total batch size of 512 (achieved with train_batch_size: 8 and gradient_accumulation_steps: 8). Key hyperparameters included a learning rate of 5e-06 and the AdamW_Torch optimizer. The training process utilized Transformers 4.46.1 and PyTorch 2.3.0.

Intended Use

Given its fine-tuning on a specific dataset, this model is best suited for applications and tasks that align closely with the characteristics and content of the mlfoundations-dev/oh_v1.3_alpaca_x2 dataset. Developers should evaluate its performance against their specific use cases to determine suitability.