AmberYifan/capsd-opc-dedup-marin-8b-base-code_cap_b8000_s0
AmberYifan/capsd-opc-dedup-marin-8b-base-code_cap_b8000_s0 is an 8 billion parameter language model, fine-tuned from marin-community/marin-8b-base. This model was trained on the capsd_marin-8b-base-n80000-opc-dedup80k__mix_code_cap_b8000_s0 dataset, suggesting a specialization in code-related tasks. It is designed for applications requiring a compact yet capable model with a context length of 8192 tokens.
Loading preview...
Model Overview
AmberYifan/capsd-opc-dedup-marin-8b-base-code_cap_b8000_s0 is an 8 billion parameter language model, fine-tuned from the existing marin-community/marin-8b-base model. This fine-tuning process utilized the capsd_marin-8b-base-n80000-opc-dedup80k__mix_code_cap_b8000_s0 dataset, indicating a potential focus on code-related applications or tasks involving specific data characteristics.
Training Details
The model underwent a single epoch of training with a learning rate of 1e-05. Key hyperparameters included a train_batch_size of 2 and an eval_batch_size of 8, with a total effective batch size of 64 due to gradient accumulation steps. The optimizer used was ADAMW_TORCH with standard beta values and epsilon, and a cosine learning rate scheduler was employed with a 0.03 warmup ratio. The training was distributed across 4 GPUs.
Technical Stack
The training environment utilized:
- Transformers 5.7.0
- Pytorch 2.13.0+cu130
- Datasets 4.0.0
- Tokenizers 0.22.2
Potential Use Cases
Given its fine-tuning on a dataset with "code_cap" and "dedup" in its name, this model is likely optimized for:
- Code generation and completion
- Code understanding and analysis
- Tasks benefiting from a model trained on deduplicated and capped code-centric data.