AmberYifan/capsd-opc-dedup-marin-8b-base-code_random_b4000_s0
AmberYifan/capsd-opc-dedup-marin-8b-base-code_random_b4000_s0 is an 8 billion parameter language model, fine-tuned from marin-community/marin-8b-base. This model was trained on a deduplicated dataset with a mix of code and random data, suggesting an optimization for code-related tasks. It utilizes a context length of 8192 tokens and was trained with a learning rate of 1e-05 over one epoch.
Loading preview...
Model Overview
This model, AmberYifan/capsd-opc-dedup-marin-8b-base-code_random_b4000_s0, is an 8 billion parameter language model. It is a fine-tuned variant of the marin-community/marin-8b-base architecture.
Key Training Details
The model was fine-tuned using the capsd_marin-8b-base-n80000-opc-dedup80k__mix_code_random_b4000_s0 dataset. The training procedure involved:
- Base Model:
marin-community/marin-8b-base - Learning Rate: 1e-05
- Batch Size: A total training batch size of 64 (with
train_batch_size: 2andgradient_accumulation_steps: 8) - Optimizer: ADAMW_TORCH
- Scheduler: Cosine learning rate scheduler
- Epochs: Trained for 1 epoch
- Context Length: Supports an 8192-token context window.
Potential Use Cases
Given its fine-tuning on a dataset described as a "mix_code_random," this model is likely optimized for tasks involving code generation, code completion, or understanding code-related contexts. Its base architecture and training on a deduplicated dataset suggest a focus on robust performance in these areas.