SKG97/qwen3-finetuned
SKG97/qwen3-finetuned is a 0.8 billion parameter language model, fine-tuned from Qwen/Qwen3-0.6B. This model was trained for 1 epoch with a learning rate of 2e-05 and a context length of 32768 tokens. It achieved a validation loss of 3.2427 on its evaluation set, indicating its performance on the specific, undisclosed fine-tuning task.
Loading preview...
Model Overview
SKG97/qwen3-finetuned is a language model based on the Qwen3-0.6B architecture, developed by SKG97. This model has 0.8 billion parameters and supports a substantial 32768-token context length, making it suitable for tasks requiring extensive contextual understanding.
Training Details
The model underwent a fine-tuning process for a single epoch using an AdamW optimizer with a learning rate of 2e-05. Key training hyperparameters included a train_batch_size of 16, eval_batch_size of 8, and gradient_accumulation_steps of 16, resulting in a total_train_batch_size of 256. During evaluation, it achieved a validation loss of 3.2427.
Technical Stack
The fine-tuning was performed using:
- Transformers 5.13.1
- Pytorch 2.11.0+cu128
- Datasets 4.0.0
- Tokenizers 0.22.2
Intended Use
While specific intended uses and limitations are not detailed in the provided information, its foundation on Qwen3-0.6B and its fine-tuned nature suggest potential application in tasks where the base model excels, adapted to the characteristics of its unknown fine-tuning dataset.