CharlesLi/llama_2_sky_safe_o1_llama_3_70B_reflect_4000_100_full
The CharlesLi/llama_2_sky_safe_o1_llama_3_70B_reflect_4000_100_full model is a 7 billion parameter language model fine-tuned from Meta's Llama-2-7b-chat-hf. This model was trained with a learning rate of 2e-05 and a batch size of 32, achieving a validation loss of 0.6944. It is intended for conversational AI applications, leveraging its Llama 2 base for enhanced dialogue capabilities.
Loading preview...
Model Overview
This model, llama_2_sky_safe_o1_llama_3_70B_reflect_4000_100_full, is a fine-tuned version of Meta's Llama-2-7b-chat-hf model. It has 7 billion parameters and was developed by CharlesLi. The fine-tuning process aimed to adapt the base Llama 2 model for specific generative tasks, as indicated by its training on a 'generator dataset'.
Training Details
The model was trained using the following key hyperparameters:
- Base Model: meta-llama/Llama-2-7b-chat-hf
- Learning Rate: 2e-05
- Total Training Batch Size: 32 (across 4 devices with 2 gradient accumulation steps)
- Optimizer: Adam with betas=(0.9, 0.999) and epsilon=1e-08
- LR Scheduler: Cosine with 0.1 warmup ratio
- Epochs: 1
During training, the model achieved a validation loss of 0.6944 after 200 steps. The training utilized Transformers 4.44.2, Pytorch 2.4.1+cu121, Datasets 3.0.0, and Tokenizers 0.19.1.
Intended Use
While specific intended uses and limitations are not detailed in the provided README, its fine-tuning from a chat-optimized base suggests its suitability for conversational AI, dialogue generation, and related natural language processing tasks.