AmberYifan/capsd-final-retrain-marin-8b-base-code_less_b8000_s0
AmberYifan/capsd-final-retrain-marin-8b-base-code_less_b8000_s0 is an 8 billion parameter language model, fine-tuned from marin-community/marin-8b-base. This model was trained on the capsd_marin-8b-base-n80000-opc__mix_code_less_b8000_s0 dataset, with a focus on reducing code content in its training mix. It is intended for general language understanding and generation tasks, with its training procedure emphasizing specific hyperparameters like a 1e-05 learning rate and 3 epochs.
Loading preview...
Model Overview
AmberYifan/capsd-final-retrain-marin-8b-base-code_less_b8000_s0 is an 8 billion parameter language model, fine-tuned from the existing marin-community/marin-8b-base model. This iteration was specifically trained on the capsd_marin-8b-base-n80000-opc__mix_code_less_b8000_s0 dataset, indicating a deliberate adjustment to the training data composition, likely with a reduced emphasis on code-related content.
Training Details
The model underwent a fine-tuning process with the following key hyperparameters:
- Learning Rate: 1e-05
- Batch Size: A
train_batch_sizeof 2 andeval_batch_sizeof 8, with atotal_train_batch_sizeof 64 (achieved with 4 devices and 8 gradient accumulation steps). - Optimizer: AdamW with default betas and epsilon.
- Scheduler: Cosine learning rate scheduler with 0.03 warmup steps.
- Epochs: Trained for 3 epochs.
Intended Use
While specific intended uses and limitations require further definition, this model is generally suitable for various natural language processing tasks. Its fine-tuning on a dataset with reduced code content suggests it may be optimized for general text generation and understanding rather than specialized code-centric applications. Developers should consider its base architecture and fine-tuning approach when evaluating its fit for specific use cases.