slayertear/llama-3.1-8b-stage1
The slayertear/llama-3.1-8b-stage1 model is an 8 billion parameter language model fine-tuned from Llama-3.1-8B-Instruct. It was specifically trained on a combined dataset of sft_chemllm (2.4M samples) and tomg (1.2M samples) chat data. This model is designed for chat-based applications, leveraging its instruction-tuned base and specialized fine-tuning for conversational tasks.
Loading preview...
Overview
This model, slayertear/llama-3.1-8b-stage1, is an 8 billion parameter language model derived from the Llama-3.1-8B-Instruct architecture. It has undergone a specific fine-tuning process to enhance its performance on conversational tasks.
Key Training Details
The model was fine-tuned using a substantial dataset comprising:
- sft_chemllm: 2.4 million samples
- tomg: 1.2 million samples
This combined dataset, totaling 3.6 million chat-based samples, indicates a strong focus on developing robust conversational abilities. The training procedure utilized a learning rate of 1e-05, a total batch size of 512 (across 8 GPUs with 4 gradient accumulation steps), and a cosine learning rate scheduler with a 0.2 warmup ratio over 3 epochs. The training leveraged Transformers 4.52.3 and PyTorch 2.6.0+cu124.
Intended Use Cases
Given its instruction-tuned base and specialized fine-tuning on chat datasets, this model is primarily intended for applications requiring:
- Conversational AI: Engaging in dialogue and generating human-like responses.
- Instruction Following: Executing commands and responding to prompts in a chat format.
Further details regarding specific intended uses, limitations, and evaluation data are noted as needing more information in the original model card.