AmberYifan/capsd-marin-8b-base-code_random_b2000_s0
AmberYifan/capsd-marin-8b-base-code_random_b2000_s0 is an 8 billion parameter language model fine-tuned from marin-community/marin-8b-base. This model was trained on the capsd_marin-8b-base-n10000__mix_code_random_b2000_s0 dataset, suggesting a specialization in code-related tasks. It utilizes a context length of 8192 tokens, making it suitable for processing moderately long code sequences.
Loading preview...
Model Overview
AmberYifan/capsd-marin-8b-base-code_random_b2000_s0 is an 8 billion parameter language model, fine-tuned from the marin-community/marin-8b-base architecture. This model was specifically trained on the capsd_marin-8b-base-n10000__mix_code_random_b2000_s0 dataset, indicating a focus on code-related applications.
Training Details
The model underwent a single epoch of training with a learning rate of 1e-05. It utilized a distributed training setup across 4 GPUs, with a total training batch size of 64 (achieved with a train_batch_size of 1 and gradient_accumulation_steps of 16). The AdamW optimizer with cosine learning rate scheduling was employed, including a 0.03 warmup ratio. The training environment included Transformers 5.7.0, Pytorch 2.13.0+cu130, Datasets 4.0.0, and Tokenizers 0.22.2.
Key Characteristics
- Base Model: Fine-tuned from
marin-community/marin-8b-base. - Parameter Count: 8 billion parameters.
- Context Length: Supports an 8192-token context window.
- Training Dataset: Specialized
capsd_marin-8b-base-n10000__mix_code_random_b2000_s0dataset, suggesting an orientation towards code generation or understanding tasks.