AmberYifan/capsd-opc-dedup-marin-8b-base-code_ppl_b20000_s0
The AmberYifan/capsd-opc-dedup-marin-8b-base-code_ppl_b20000_s0 model is an 8 billion parameter language model, fine-tuned from marin-community/marin-8b-base. It was trained on a specialized dataset, capsd_marin-8b-base-n80000-opc-dedup80k__mix_code_ppl_b20000_s0, suggesting an optimization for code-related tasks. This model is designed for applications requiring a base language model with potential enhancements for code understanding or generation, leveraging its 8192 token context length.
Loading preview...
Model Overview
This model, AmberYifan/capsd-opc-dedup-marin-8b-base-code_ppl_b20000_s0, is an 8 billion parameter language model. It is a fine-tuned version of the marin-community/marin-8b-base architecture, indicating a foundation in a pre-existing base model.
Training Details
The model underwent a fine-tuning process using the capsd_marin-8b-base-n80000-opc-dedup80k__mix_code_ppl_b20000_s0 dataset. Key training hyperparameters included:
- Learning Rate: 1e-05
- Batch Size: 2 (train), 8 (eval)
- Gradient Accumulation Steps: 8, leading to a total train batch size of 64
- Optimizer: ADAMW_TORCH
- LR Scheduler: Cosine type with 0.03 warmup steps
- Epochs: 1
Framework Versions
The training utilized:
- Transformers 5.7.0
- Pytorch 2.13.0+cu130
- Datasets 4.0.0
- Tokenizers 0.22.2
Potential Use Cases
Given its fine-tuning on a dataset with "code" and "dedup" in its name, this model is likely optimized for tasks involving code processing, generation, or understanding. Its 8B parameters and 8192 token context length make it suitable for handling moderately complex code-related queries.