AmberYifan/capsd-marin-8b-base-science_cap_b4000_s0
The AmberYifan/capsd-marin-8b-base-science_cap_b4000_s0 is an 8 billion parameter language model, fine-tuned from marin-community/marin-8b-base. This model was trained on a specific dataset, capsd_marin-8b-base-n10000__mix_science_cap_b4000_s0, suggesting a specialization in scientific or technical domains. With an 8192 token context length, it is designed for tasks requiring processing moderately long inputs related to its fine-tuning data.
Loading preview...
Model Overview
This model, AmberYifan/capsd-marin-8b-base-science_cap_b4000_s0, is an 8 billion parameter language model. It is a fine-tuned variant of the marin-community/marin-8b-base architecture, specifically adapted through further training.
Training Details
The model underwent fine-tuning using the capsd_marin-8b-base-n10000__mix_science_cap_b4000_s0 dataset. Key training hyperparameters included:
- Learning Rate: 1e-05
- Batch Size: 1 (train), 8 (eval)
- Gradient Accumulation Steps: 16, leading to a total effective batch size of 64
- Optimizer: AdamW with default betas and epsilon
- LR Scheduler: Cosine type with 0.03 warmup steps
- Epochs: 1
Frameworks Used
The training process utilized:
- Transformers 5.7.0
- Pytorch 2.13.0+cu130
- Datasets 4.0.0
- Tokenizers 0.22.2
Current Status
As per the model card, more detailed information regarding the model's description, intended uses, limitations, and specific training/evaluation data is currently needed. Users should be aware that comprehensive documentation is still pending.