tsavage68/chat_400STEPS_1e6rate_SFT
tsavage68/chat_400STEPS_1e6rate_SFT is a 7 billion parameter Llama-2-7b-chat-hf based model, fine-tuned for chat applications. This model was trained with a low learning rate of 1e-06 over 400 steps, achieving a final validation loss of 0.3202. It is intended for general conversational tasks, leveraging its Llama 2 foundation.
Loading preview...
Model Overview
tsavage68/chat_400STEPS_1e6rate_SFT is a 7 billion parameter language model derived from meta-llama/Llama-2-7b-chat-hf. This model has undergone supervised fine-tuning (SFT) with a specific training regimen, focusing on stability and convergence with a very low learning rate.
Training Details
The model was fine-tuned over 400 training steps using a learning rate of 1e-06, a train_batch_size of 4, and gradient_accumulation_steps of 2, resulting in an effective total batch size of 8. The optimizer used was Adam, and the learning rate scheduler was set to cosine with 100 warmup steps. During training, the validation loss steadily decreased, reaching a final value of 0.3202 at step 400.
Key Characteristics
- Base Model: Llama-2-7b-chat-hf
- Parameter Count: 7 billion
- Training Steps: 400
- Learning Rate: 1e-06
- Final Validation Loss: 0.3202
Intended Use Cases
This model is suitable for general chat and conversational AI applications, building upon the robust capabilities of its Llama 2 base. Its fine-tuning process suggests an emphasis on refining conversational fluency and response quality within its training domain, though specific details on the fine-tuning dataset are not provided.