Model Overview

This model, llama_2_sky_safe_o1_4o_reflect_4000_1000_full, is a fine-tuned version of the meta-llama/Llama-2-7b-chat-hf base model, developed by CharlesLi. It features 7 billion parameters and was trained for a single epoch with a learning rate of 2e-05.

Training Details

The model was trained using the following key hyperparameters:

• Learning Rate: 2e-05
• Batch Sizes: train_batch_size of 4, eval_batch_size of 4
• Optimizer: Adam with default betas and epsilon
• LR Scheduler: Cosine type with a warmup ratio of 0.1
• Epochs: 1

During training, a validation loss of 0.5639 was observed at step 100, with a final training loss of 0.5360. The training utilized a multi-GPU setup with 4 devices and a gradient accumulation of 2 steps.

Framework Versions

• Transformers: 4.44.2
• Pytorch: 2.4.1+cu121
• Datasets: 3.0.0
• Tokenizers: 0.19.1

Intended Uses

Based on its Llama 2 Chat foundation, this model is likely suitable for general conversational AI, instruction following, and text generation tasks, though specific optimizations or unique capabilities are not detailed in the provided information.