VladShash/qwen3-4b-feedback-one-shot
VladShash/qwen3-4b-feedback-one-shot is a 4 billion parameter language model fine-tuned from formalmathatepfl/qwen3-4b-cpt. This model is specifically fine-tuned on the 'sft' dataset, indicating a focus on supervised fine-tuning tasks. With a context length of 32768 tokens, it is designed for applications requiring processing of longer sequences. Its primary differentiation lies in its specific fine-tuning approach, making it suitable for tasks aligned with the 'sft' dataset's characteristics.
Loading preview...
Model Overview
VladShash/qwen3-4b-feedback-one-shot is a 4 billion parameter language model derived from formalmathatepfl/qwen3-4b-cpt. This model has undergone supervised fine-tuning (SFT) using a specific 'sft' dataset, which suggests its optimization for tasks related to that dataset's domain. It supports a substantial context length of 32768 tokens, enabling it to handle extensive textual inputs and generate coherent long-form outputs.
Training Details
The model was trained with a learning rate of 2e-05, a batch size of 1 per device across 8 GPUs, totaling an effective batch size of 8. The training utilized the AdamW optimizer with cosine learning rate scheduling and a warmup ratio of 0.05 over 1 epoch. This configuration indicates a focused fine-tuning process aimed at adapting the base model to specific SFT objectives.
Key Characteristics
- Base Model: Fine-tuned from
formalmathatepfl/qwen3-4b-cpt. - Parameter Count: 4 billion parameters.
- Context Length: 32768 tokens.
- Fine-tuning: Supervised fine-tuning on an 'sft' dataset.
Potential Use Cases
Given its SFT nature and the base model, this model is likely suitable for tasks that align with the 'sft' dataset it was trained on. Developers should evaluate its performance for specific applications requiring a 4B parameter model with extended context capabilities.