robertvacareanu/instruct-rl-Qwen2.5-0.5B-databricks-dolly-15k
The robertvacareanu/instruct-rl-Qwen2.5-0.5B-databricks-dolly-15k is a 0.5 billion parameter instruction-tuned language model, fine-tuned from Qwen/Qwen2.5-0.5B. This model is specifically adapted for instruction following, leveraging the Databricks Dolly 15k dataset. It is designed for efficient deployment in applications requiring a compact yet capable model for general instruction-based tasks.
Loading preview...
Model Overview
This model, instruct-rl-Qwen2.5-0.5B-databricks-dolly-15k, is a fine-tuned variant of the Qwen/Qwen2.5-0.5B base model. It has been specifically adapted for instruction-following capabilities through training on the Databricks Dolly 15k dataset.
Key Characteristics
- Base Model: Qwen/Qwen2.5-0.5B
- Parameter Count: 0.5 billion parameters, making it a compact and efficient choice.
- Context Length: Supports a substantial context window of 32768 tokens.
- Fine-tuning Dataset: Utilizes the Databricks Dolly 15k dataset, known for its human-generated instruction-following examples.
Training Details
The model was trained with a learning rate of 5e-05, a total batch size of 32 (achieved with train_batch_size: 8 and gradient_accumulation_steps: 4), and for 1 epoch. The optimizer used was Adam with standard betas and epsilon, and a linear learning rate scheduler with a 0.1 warmup ratio.
Intended Use
This model is suitable for applications requiring a small, instruction-tuned language model capable of understanding and responding to various prompts based on the Dolly 15k dataset's instruction format. Its compact size makes it ideal for resource-constrained environments or edge deployments where larger models are impractical.