AmberYifan/capsd-marin-8b-base-code_random_b12000_s0
AmberYifan/capsd-marin-8b-base-code_random_b12000_s0 is an 8 billion parameter language model fine-tuned from marin-community/marin-8b-base. This model was trained on the capsd_marin-8b-base-n80000-opc__mix_code_random_b12000_s0 dataset, indicating a specialization in code-related tasks. It utilizes a cosine learning rate scheduler and was trained for one epoch with a learning rate of 1e-05. This model is intended for applications requiring code generation or understanding, leveraging its base architecture and specific fine-tuning.
Loading preview...
Model Overview
AmberYifan/capsd-marin-8b-base-code_random_b12000_s0 is an 8 billion parameter language model that has been fine-tuned from the marin-community/marin-8b-base architecture. The fine-tuning process specifically utilized the capsd_marin-8b-base-n80000-opc__mix_code_random_b12000_s0 dataset, suggesting an optimization for tasks involving code.
Training Details
The model was trained using the following key hyperparameters:
- Learning Rate: 1e-05
- Batch Size: A
train_batch_sizeof 2 andeval_batch_sizeof 8, leading to atotal_train_batch_sizeof 64 andtotal_eval_batch_sizeof 32, with 8 gradient accumulation steps. - Optimizer: ADAMW_TORCH with default betas and epsilon.
- LR Scheduler: Cosine type with 0.03 warmup steps.
- Epochs: Trained for 1 epoch.
This configuration indicates a focused training approach aimed at enhancing the model's capabilities on the specific fine-tuning dataset. The model was developed using Transformers 5.7.0, Pytorch 2.13.0+cu130, Datasets 4.0.0, and Tokenizers 0.22.2.