AmberYifan/capsd-less-ultra-opc-marin-8b-base-code_less_b1000_s0
AmberYifan/capsd-less-ultra-opc-marin-8b-base-code_less_b1000_s0 is an 8 billion parameter language model fine-tuned from marin-community/marin-8b-base. This model was trained on a specific dataset, capsd_marin-8b-base-n80000-opc__mix_code_less_b1000_s0, with a focus on code-related tasks. It utilizes an 8192 token context length and is intended for applications requiring a specialized base model for code processing.
Loading preview...
Model Overview
This model, AmberYifan/capsd-less-ultra-opc-marin-8b-base-code_less_b1000_s0, is an 8 billion parameter language model derived from marin-community/marin-8b-base. It has been fine-tuned with a specific dataset, capsd_marin-8b-base-n80000-opc__mix_code_less_b1000_s0, indicating a specialization towards code-related tasks, though further details on its exact capabilities are not provided in the original model card.
Training Details
The model was trained using the following key hyperparameters:
- Learning Rate: 1e-05
- Batch Size: 2 (train), 8 (eval)
- Gradient Accumulation Steps: 8, leading to a total effective batch size of 64
- Optimizer: ADAMW_TORCH
- LR Scheduler: Cosine with 0.03 warmup steps
- Epochs: 1
It was trained on 4 GPUs using a multi-GPU distributed setup. The training utilized Transformers 5.7.0, Pytorch 2.13.0+cu130, Datasets 4.0.0, and Tokenizers 0.22.2.
Intended Use
While specific use cases are not detailed, its fine-tuning on a code-centric dataset suggests its potential for applications involving code generation, completion, or analysis. Users should be aware that the model card indicates a need for more information regarding its intended uses and limitations.