shirley09221/qwen3-8b-full-pretrain-junk-tweet-1m-en-sft
The shirley09221/qwen3-8b-full-pretrain-junk-tweet-1m-en-sft model is an 8 billion parameter language model, fine-tuned from shirley09221/qwen3-8b-full-pretrain-junk-tweet-1m-en. It was further trained on the alpaca_en dataset, suggesting a focus on instruction-following capabilities. With a context length of 32768 tokens, this model is designed for general text generation and understanding tasks, particularly those benefiting from instruction tuning.
Loading preview...
Model Overview
This model, shirley09221/qwen3-8b-full-pretrain-junk-tweet-1m-en-sft, is an 8 billion parameter language model. It is a fine-tuned variant of the qwen3-8b-full-pretrain-junk-tweet-1m-en base model, specifically adapted through further training on the alpaca_en dataset. This fine-tuning process typically enhances a model's ability to follow instructions and generate coherent, task-specific responses.
Training Details
The model was trained using the following key hyperparameters:
- Learning Rate: 1e-05
- Batch Size: 1 (train), 8 (eval)
- Gradient Accumulation Steps: 2, leading to a total train batch size of 16
- Optimizer: AdamW_Torch with default betas and epsilon
- LR Scheduler: Cosine with 0.1 warmup steps
- Epochs: 3.0
The training was conducted across 8 GPUs, utilizing a multi-GPU distributed setup. The development environment included Transformers 5.8.0, Pytorch 2.13.0+cu130, Datasets 4.0.0, and Tokenizers 0.22.2.
Potential Use Cases
Given its fine-tuning on the alpaca_en dataset, this model is likely suitable for tasks requiring:
- Instruction following
- General text generation based on prompts
- Conversational AI or chatbot applications where instruction adherence is key