AmberYifan/capsd-opc-dedup-marin-8b-base-code_ppl_b8000_s0
The AmberYifan/capsd-opc-dedup-marin-8b-base-code_ppl_b8000_s0 is an 8 billion parameter language model, fine-tuned from marin-community/marin-8b-base. This model was trained on a specific dataset, capsd_marin-8b-base-n80000-opc-dedup80k__mix_code_ppl_b8000_s0, suggesting an optimization for code-related tasks. With a context length of 8192 tokens, it is designed for applications requiring processing of moderately long sequences.
Loading preview...
Model Overview
This model, AmberYifan/capsd-opc-dedup-marin-8b-base-code_ppl_b8000_s0, is an 8 billion parameter language model. It is a fine-tuned variant of the marin-community/marin-8b-base architecture, specifically adapted through further training.
Training Details
The model was fine-tuned on the capsd_marin-8b-base-n80000-opc-dedup80k__mix_code_ppl_b8000_s0 dataset. Key training hyperparameters included:
- Learning Rate: 1e-05
- Batch Size: 2 (train), 8 (eval)
- Gradient Accumulation Steps: 8, leading to a total effective train batch size of 64
- Optimizer: ADAMW_TORCH with default betas and epsilon
- LR Scheduler: Cosine type with 0.03 warmup steps
- Epochs: 1
The training utilized a multi-GPU setup with 4 devices. The development environment included Transformers 5.7.0, Pytorch 2.13.0+cu130, Datasets 4.0.0, and Tokenizers 0.22.2.
Potential Use Cases
Given its fine-tuning on a dataset with "code" in its name, this model is likely optimized for tasks involving code generation, completion, analysis, or understanding. Its 8B parameter count and 8192 token context window make it suitable for applications requiring processing of code snippets or related textual data.