[

](https://github.com/axolotl-ai-cloud/axolotl)

See axolotl config

axolotl version: `0.8.0` ```yaml base_model: Qwen/Qwen2.5-Coder-14B strict: false plugins: - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin - axolotl.integrations.liger.LigerPlugin cut_cross_entropy: true # liger_rope: true liger_rms_norm: true liger_layer_norm: true # gemma3 doesn't seem to play nice with ddp ddp_find_unused_parameters: true chat_template: qwen_25 datasets: - path: winglian/gpumode-py2triton-reasoning type: chat_template dataset_prepared_path: last_run_prepared val_set_size: 0.0 output_dir: ./outputs/out-fft save_safetensors: true save_only_model: true sequence_len: 32768 sample_packing: true pad_to_sequence_len: true sequence_parallel_degree: 1 # unfrozen_parameters: # - language_model.model wandb_project: qwen25-kernel-llm wandb_entity: axolotl-ai wandb_watch: wandb_name: wandb_log_model: gradient_accumulation_steps: 1 micro_batch_size: 3 num_epochs: 3 optimizer: adamw_8bit lr_scheduler: rex learning_rate: 3.0e-6 lr_groups: - name: embeddings lr: 3.0e-5 modules: - lm_head - embed_tokens train_on_inputs: false group_by_length: false bf16: true tf32: true gradient_checkpointing: true gradient_checkpointing_kwargs: use_reentrant: false logging_steps: 1 flash_attention: true warmup_ratio: 0.1 evals_per_epoch: 1 saves_per_epoch: 1 # deepspeed: deepspeed_configs/zero1.json weight_decay: 0.0 deepspeed: deepspeed_configs/zero1.json tokens: - - special_tokens: eos_token: <|im_end|> fix_untrained_tokens: - 151665 - 151666 ```

# outputs/out-fft This model is a fine-tuned version of [Qwen/Qwen2.5-Coder-14B](https://huggingface.co/Qwen/Qwen2.5-Coder-14B) on the winglian/gpumode-py2triton-reasoning dataset. ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 3e-06 - train_batch_size: 3 - eval_batch_size: 3 - seed: 42 - distributed_type: multi-GPU - num_devices: 8 - total_train_batch_size: 24 - total_eval_batch_size: 24 - optimizer: Use OptimizerNames.ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments - lr_scheduler_type: cosine - lr_scheduler_warmup_steps: 27 - num_epochs: 3.0 ### Training results ### Framework versions - Transformers 4.51.3 - Pytorch 2.6.0+cu124 - Datasets 3.5.0 - Tokenizers 0.21.1