Model Overview

CharlesLi/llama_2_sky_safe_o1_4o_reflect_4000_500_full is a 7 billion parameter language model derived from Meta's Llama-2-7b-chat-hf. This model has undergone a specific fine-tuning process, evidenced by its reported evaluation loss of 0.6698.

Training Details

The fine-tuning procedure utilized the following key hyperparameters:

• Base Model: meta-llama/Llama-2-7b-chat-hf
• Learning Rate: 2e-05
• Optimizer: Adam with betas=(0.9, 0.999) and epsilon=1e-08
• Scheduler: Cosine learning rate scheduler with a warmup ratio of 0.1
• Epochs: 1
• Total Batch Size: 32 (across 4 GPUs with 2 gradient accumulation steps)

Training results show a progressive reduction in loss, with a final validation loss of 0.6827 at 200 steps. The model was trained using Transformers 4.44.2, Pytorch 2.4.1+cu121, Datasets 3.0.0, and Tokenizers 0.19.1.

Capabilities & Limitations

As a fine-tuned variant of Llama-2-7b-chat-hf, this model likely retains strong general-purpose conversational and text generation abilities. However, specific intended uses, detailed capabilities, and known limitations are not explicitly provided in the current documentation. Users should refer to the base Llama 2 model's documentation for general characteristics and conduct further evaluation for specific applications.