MarisUK/master
MarisUK/master is a 0.5 billion parameter causal language model, fine-tuned from Qwen/Qwen2.5-0.5B-Instruct. This model is specifically adapted using a generator dataset, achieving a validation loss of 2.5193. It is designed for tasks benefiting from instruction-tuned Qwen2.5 architecture in a compact size.
Loading preview...
Model Overview
MarisUK/master is a compact 0.5 billion parameter language model, fine-tuned from the Qwen/Qwen2.5-0.5B-Instruct base model. This fine-tuning process utilized a specific "generator dataset" to adapt its capabilities. During training, the model achieved a validation loss of 2.5193, indicating its performance on the evaluation set.
Training Details
The model was trained with a learning rate of 2e-05 over 1 epoch, using a batch size of 1 and accumulating gradients over 8 steps, resulting in an effective total batch size of 8. The AdamW_TORCH_FUSED optimizer was employed, and the learning rate schedule followed a cosine decay with 0.1 warmup steps. The training was conducted using Transformers 5.5.3, Pytorch 2.11.0+cu130, Datasets 4.8.4, and Tokenizers 0.22.2.
Key Characteristics
- Base Model: Qwen/Qwen2.5-0.5B-Instruct
- Parameter Count: 0.5 billion
- Context Length: 32768 tokens
- Fine-tuning: Specialized on a "generator dataset"
- Performance Metric: Achieved a validation loss of 2.5193
Potential Use Cases
Given its compact size and instruction-tuned base, this model is suitable for applications requiring efficient inference and deployment on resource-constrained environments. Its fine-tuning on a generator dataset suggests potential for tasks involving text generation or response formulation within its learned domain.