AmberYifan/capsd-marin-8b-base-code_qurating_b10000_s0

Model Overview

AmberYifan/capsd-marin-8b-base-code_qurating_b10000_s0 is an 8 billion parameter language model, fine-tuned from the marin-community/marin-8b-base architecture. This model has been specifically trained on the capsd_marin-8b-base-n80000-opc__mix_code_qurating_b10000_s0 dataset, suggesting a specialization in code-related applications.

Key Training Details

The model underwent a fine-tuning process with the following hyperparameters:

• Learning Rate: 1e-05
• Batch Size: A train_batch_size of 2 and eval_batch_size of 8 were used, with a total_train_batch_size of 64 and total_eval_batch_size of 32 due to gradient accumulation.
• Optimizer: ADAMW_TORCH with default betas and epsilon.
• Scheduler: Cosine learning rate scheduler with 0.03 warmup steps.
• Epochs: Trained for 1 epoch.
• Environment: Training was conducted on a multi-GPU setup with 4 devices.

Framework Versions

The training utilized:

• Transformers 5.7.0
• Pytorch 2.13.0+cu130
• Datasets 4.0.0
• Tokenizers 0.22.2

Intended Use

Given its fine-tuning on a code-centric dataset, this model is likely intended for tasks involving code generation, completion, analysis, or understanding. Further details on specific use cases and limitations are not provided in the current model card.