[

](https://github.com/axolotl-ai-cloud/axolotl)

See axolotl config

axolotl version: `0.4.1` ```yaml base_model: meta-llama/Llama-3.2-1B model_type: LlamaForCausalLM tokenizer_type: AutoTokenizer load_in_8bit: false load_in_4bit: false strict: false chat_template: chatml datasets: - path: datasets/airoboros_3.2_without_contextual_slimorca_orca_sharegpt.json ds_type: json type: sharegpt conversation: chatml - path: datasets/allenai_wild_chat_gpt4_english_toxic_random_half_4k_sharegpt.json ds_type: json type: sharegpt strict: false conversation: chatml - path: datasets/buzz_unstacked_chosen_math_removed_filtered.json ds_type: json type: alpaca conversation: chatml - path: datasets/capybara_sharegpt.json ds_type: json type: sharegpt conversation: chatml - path: datasets/cot_alpaca_gpt4_extracted_openhermes_2.5_sharegpt.json ds_type: json type: sharegpt conversation: chatml - path: datasets/everythinglm-data-v3_sharegpt.json ds_type: json type: sharegpt strict: false conversation: chatml - path: datasets/gpt4_data_lmys_1m_sharegpt.json ds_type: json type: sharegpt conversation: chatml - path: datasets/gpteacher-instruct-special-alpaca.json ds_type: json type: gpteacher conversation: chatml - path: datasets/merged_all.json ds_type: json type: alpaca conversation: chatml - path: datasets/no_robots_sharegpt.json ds_type: json type: sharegpt strict: false conversation: chatml - path: datasets/oasst_top1_from_fusechatmixture_sharegpt.json ds_type: json type: sharegpt strict: false conversation: chatml - path: datasets/pippa_bagel_repo_3k_sharegpt.json ds_type: json type: sharegpt conversation: chatml - path: datasets/rpguild_quarter_alignment_lab_sharegpt.json ds_type: json type: sharegpt conversation: chatml - path: datasets/sharegpt_gpt4_english.json ds_type: json type: sharegpt conversation: chatml - path: datasets/slimorca_dedup_filtered_95k_sharegpt.json ds_type: json type: sharegpt conversation: chatml - path: datasets/soda_diaolog_longest_tenth_buzz_sharegpt.json ds_type: json type: sharegpt conversation: chatml - path: datasets/synthia-v1.3_sharegpt_12500.json ds_type: json type: sharegpt conversation: chatml - path: datasets/system_conversations_dolphin_sharegpt.json ds_type: json type: sharegpt conversation: chatml - path: datasets/NuminaMath-CoT-olympiads-40k_alpaca.json ds_type: json type: alpaca conversation: chatml - path: datasets/math-gpt-4o-40k_alpaca.json ds_type: json type: alpaca conversation: chatml - path: datasets/sonnet3.5_science_conversations_sharegpt.json ds_type: json type: sharegpt conversation: chatml - path: datasets/reasoning-0.01_sharegpt.jsonl ds_type: json type: sharegpt conversation: chatml dataset_prepared_path: last_run_prepared val_set_size: 0.002 output_dir: ./Einstein-v8-Llama3.2-1B-model sequence_len: 8192 sample_packing: true pad_to_sequence_len: true eval_sample_packing: false wandb_project: Einstein wandb_entity: wandb_watch: wandb_name: Einstein-v8-Llama3.2-1B-2-epoch wandb_log_model: hub_model_id: Weyaxi/Einstein-v8-Llama3.2-1B save_safetensors: true gradient_accumulation_steps: 4 micro_batch_size: 4 num_epochs: 2 optimizer: adamw_bnb_8bit # look lr_scheduler: cosine learning_rate: 0.000005 # look train_on_inputs: false group_by_length: false bf16: true fp16: false tf32: false gradient_checkpointing: true early_stopping_patience: resume_from_checkpoint: local_rank: logging_steps: 1 xformers_attention: flash_attention: true warmup_steps: 10 evals_per_epoch: 4 eval_table_size: eval_table_max_new_tokens: 128 saves_per_epoch: 1 debug: deepspeed: axolotl/deepspeed_configs/zero3_bf16.json weight_decay: 0.0 fsdp: fsdp_config: special_tokens: bos_token: "~~" eos_token: "<|im_end|>" unk_token: "" pad_token: <|end_of_text|> # changed tokens: - "<|im_start|>" ```~~

# Einstein-v8-Llama3.2-1B This model is a fine-tuned version of [meta-llama/Llama-3.2-1B](https://huggingface.co/meta-llama/Llama-3.2-1B) on the None dataset. It achieves the following results on the evaluation set: - Loss: 0.9292 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 5e-06 - train_batch_size: 4 - eval_batch_size: 4 - seed: 42 - distributed_type: multi-GPU - num_devices: 4 - gradient_accumulation_steps: 4 - total_train_batch_size: 64 - total_eval_batch_size: 16 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: cosine - lr_scheduler_warmup_steps: 10 - num_epochs: 2 ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:------:|:----:|:---------------:| | 1.4261 | 0.0009 | 1 | 1.4028 | | 1.0487 | 0.2501 | 268 | 0.9917 | | 1.0484 | 0.5001 | 536 | 0.9652 | | 1.0039 | 0.7502 | 804 | 0.9499 | | 1.0528 | 1.0002 | 1072 | 0.9399 | | 0.9559 | 1.2481 | 1340 | 0.9345 | | 0.9078 | 1.4981 | 1608 | 0.9309 | | 0.9702 | 1.7481 | 1876 | 0.9295 | | 0.929 | 1.9981 | 2144 | 0.9292 | ### Framework versions - Transformers 4.45.0 - Pytorch 2.3.1+cu121 - Datasets 2.21.0 - Tokenizers 0.20.0 # [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_Weyaxi__Einstein-v8-Llama3.2-1B) | Metric |Value| |-------------------|----:| |Avg. | 4.63| |IFEval (0-Shot) |18.62| |BBH (3-Shot) | 3.01| |MATH Lvl 5 (4-Shot)| 0.00| |GPQA (0-shot) | 1.12| |MuSR (0-shot) | 3.22| |MMLU-PRO (5-shot) | 1.79|