AmberYifan/capsd-marin-8b-base-code_ifd_b14000_s0
AmberYifan/capsd-marin-8b-base-code_ifd_b14000_s0 is an 8 billion parameter language model, fine-tuned from marin-community/marin-8b-base. This model was trained on the capsd_marin-8b-base-n80000-opc__mix_code_ifd_b14000_s0 dataset, suggesting a specialization in code-related tasks. It features an 8192-token context length and was trained using specific hyperparameters including a learning rate of 1e-05 and a cosine learning rate scheduler.
Loading preview...
Model Overview
AmberYifan/capsd-marin-8b-base-code_ifd_b14000_s0 is an 8 billion parameter language model, fine-tuned from the marin-community/marin-8b-base architecture. This model is specifically adapted through further training on the capsd_marin-8b-base-n80000-opc__mix_code_ifd_b14000_s0 dataset, indicating a potential focus on code-related applications.
Key Training Details
The model was trained with the following hyperparameters:
- Learning Rate: 1e-05
- Optimizer: ADAMW_TORCH with betas=(0.9, 0.999) and epsilon=1e-08
- LR Scheduler: Cosine type with 0.03 warmup steps
- Batch Size: A total training batch size of 64 (2 per device across 4 GPUs with 8 gradient accumulation steps)
- Epochs: 1
Intended Use
While specific intended uses and limitations are not detailed in the provided information, the fine-tuning on a dataset with "code" in its name suggests its utility for tasks involving code generation, understanding, or analysis. Developers looking for a base model potentially optimized for programming-centric applications might consider this model.