phunguyen01/Llama-Tulu-8B-SFT
phunguyen01/Llama-Tulu-8B-SFT is an 8 billion parameter causal language model, fine-tuned from Meta's Llama-3.1-8B architecture. It was trained on the allenai/tulu-3-sft-mixture dataset, specializing in instruction-following tasks. This model is designed for general-purpose conversational AI and instruction-based applications, leveraging a 32768 token context length for enhanced understanding and generation.
Loading preview...
Model Overview
phunguyen01/Llama-Tulu-8B-SFT is an 8 billion parameter instruction-tuned language model, built upon the robust meta-llama/Llama-3.1-8B base architecture. This model has been fine-tuned using the allenai/tulu-3-sft-mixture dataset, which is designed to enhance its capabilities in following instructions and engaging in conversational tasks.
Key Characteristics
- Base Model: Fine-tuned from Meta's Llama-3.1-8B.
- Dataset: Utilizes the
allenai/tulu-3-sft-mixturefor supervised fine-tuning (SFT). - Context Length: Supports a sequence length of 2048 tokens during training, with a reported context length of 32768 tokens.
- Training Framework: Developed using Axolotl, a popular framework for LLM training.
Training Details
The model was trained with a learning rate of 5e-06 over 2 epochs, using a total batch size of 32 across 8 GPUs. Flash Attention was enabled during training to optimize performance. The training procedure included a cosine learning rate scheduler with 100 warmup steps.
Intended Use Cases
This model is suitable for a variety of instruction-following applications, including:
- General-purpose chatbots.
- Question answering systems.
- Content generation based on specific prompts.
- Tasks requiring adherence to given instructions.