Model Overview

This model, llama_2_sky_safe_o1_4o_default_4000_1000_full, is a fine-tuned version of Meta's Llama-2-7b-chat-hf model. Developed by CharlesLi, it leverages the 7 billion parameter Llama 2 architecture.

Training Details

The model was trained using the following hyperparameters:

• Learning Rate: 2e-05
• Batch Size: 4 (train), 4 (eval)
• Gradient Accumulation Steps: 2
• Optimizer: Adam with betas=(0.9, 0.999) and epsilon=1e-08
• LR Scheduler: Cosine with 0.1 warmup ratio
• Epochs: 1

During training, it achieved a validation loss of 0.5409. The training process utilized a multi-GPU setup with 4 devices, resulting in a total train batch size of 32 and a total eval batch size of 16.

Limitations

The provided information does not specify the dataset used for fine-tuning, nor does it detail the intended uses, specific capabilities, or known limitations of this particular fine-tuned version. Further information is needed to understand its unique strengths or ideal applications.