AmberYifan/capsd-opc-dedup-marin-8b-base-code_random_b2000_s0
The AmberYifan/capsd-opc-dedup-marin-8b-base-code_random_b2000_s0 is an 8 billion parameter language model, fine-tuned from marin-community/marin-8b-base. This model was specifically fine-tuned on a dataset combining capsd_marin-8b-base-n80000-opc-dedup80k with a mix of code-related data. It is designed for tasks benefiting from its specialized training on code and deduplicated datasets, offering a context length of 8192 tokens.
Loading preview...
Model Overview
This model, AmberYifan/capsd-opc-dedup-marin-8b-base-code_random_b2000_s0, is an 8 billion parameter language model. It is a fine-tuned variant of the marin-community/marin-8b-base model, specifically adapted through further training.
Key Training Details
The model was fine-tuned using a specialized dataset named capsd_marin-8b-base-n80000-opc-dedup80k__mix_code_random_b2000_s0. This indicates a focus on a deduplicated dataset combined with a mix of code-related data, suggesting an optimization for tasks involving code or requiring robust data processing.
Training was conducted with the following hyperparameters:
- Learning Rate: 1e-05
- Optimizer: ADAMW_TORCH with betas=(0.9, 0.999) and epsilon=1e-08
- Batch Size: A total train batch size of 64 (2 per device across 4 GPUs with 8 gradient accumulation steps)
- Scheduler: Cosine learning rate scheduler with 0.03 warmup steps
- Epochs: 1
Framework Versions
The training environment utilized:
- Transformers 5.7.0
- Pytorch 2.13.0+cu130
- Datasets 4.0.0
- Tokenizers 0.22.2
Potential Use Cases
Given its fine-tuning on a code-mixed and deduplicated dataset, this model is likely suitable for applications requiring:
- Code understanding or generation tasks.
- Processing and generating text from large, potentially redundant datasets where deduplication is beneficial.
- Tasks where the base
marin-8b-basemodel's capabilities are enhanced by specialized code training.