Model Overview

CharlesLi/llama_2_alpaca_helpful is a 7 billion parameter language model derived from Meta's Llama-2-7b-chat-hf architecture. This model has undergone a specific fine-tuning process, though the exact dataset used for this fine-tuning is not specified in the provided information.

Training Details

The model was trained using the following key hyperparameters:

• Learning Rate: 0.0002
• Batch Size: 4 (train and eval), with a total effective batch size of 16 due to gradient accumulation
• Optimizer: Adam with betas=(0.9, 0.999) and epsilon=1e-08
• Scheduler: Cosine learning rate scheduler with a 0.1 warmup ratio
• Training Steps: 50 steps across 2 GPUs

During training, the model achieved a final validation loss of 0.7725. The context length for this model is 4096 tokens.

Intended Use

While specific intended uses and limitations are not detailed, as a fine-tuned variant of a chat-optimized Llama 2 model, it is generally suitable for conversational AI applications where helpful responses are desired.