adpretko/armv8-o2-full-qwen-epoch1-AMD
The adpretko/armv8-o2-full-qwen-epoch1-AMD model is a 1.5 billion parameter language model, fine-tuned from a checkpoint of saves/armv8-o2-full-qwen-epoch1-AMD/checkpoint-1200. This model was trained with a context length of 32768 tokens, utilizing a cosine learning rate scheduler and AdamW optimizer. Its primary characteristic is being a fine-tuned Qwen-based model, though specific differentiators beyond its training configuration are not detailed in the provided information.
Loading preview...
Model Overview
This model, adpretko/armv8-o2-full-qwen-epoch1-AMD, is a 1.5 billion parameter language model. It is a fine-tuned version derived from a checkpoint of saves/armv8-o2-full-qwen-epoch1-AMD/checkpoint-1200.
Training Details
The model was trained using the following key hyperparameters:
- Learning Rate: 2e-05
- Batch Size: A total training batch size of 512 (with
train_batch_size: 8andgradient_accumulation_steps: 8across 8 devices). - Optimizer: AdamW_TORCH_FUSED with betas=(0.9, 0.999) and epsilon=1e-08.
- Scheduler: Cosine learning rate scheduler with a warmup ratio of 0.1.
- Epochs: Trained for 1.0 epoch.
Framework Versions
The training environment utilized:
- Transformers 4.55.0
- Pytorch 2.8.0+rocm6.3
- Datasets 3.6.0
- Tokenizers 0.21.1
Limitations
Specific intended uses, limitations, and details about the training and evaluation datasets are not provided in the available model information. Users should exercise caution and conduct further evaluation for specific applications.