AmberYifan/capsdnum-marin-8b-base-code_random_b4000_s0
AmberYifan/capsdnum-marin-8b-base-code_random_b4000_s0 is an 8 billion parameter language model, fine-tuned from marin-community/marin-8b-base. This model was trained on a specific dataset, capsd_marin-8b-base-n80000-opc__mix_code_random_b4000_s0, suggesting a specialization in code-related tasks. It utilizes a context length of 8192 tokens and was trained with a learning rate of 1e-05 over one epoch.
Loading preview...
Model Overview
AmberYifan/capsdnum-marin-8b-base-code_random_b4000_s0 is an 8 billion parameter language model, fine-tuned from the marin-community/marin-8b-base architecture. This model was specifically trained on the capsd_marin-8b-base-n80000-opc__mix_code_random_b4000_s0 dataset, indicating a potential focus on code-related applications or data.
Training Details
The model underwent a single training epoch with a learning rate of 1e-05. Key hyperparameters included a train_batch_size of 2, eval_batch_size of 8, and a gradient_accumulation_steps of 8, resulting in a total_train_batch_size of 64. The optimizer used was ADAMW_TORCH with standard betas and epsilon, and a cosine learning rate scheduler was employed. The training utilized a multi-GPU setup with 4 devices.
Key Characteristics
- Base Model: Fine-tuned from
marin-community/marin-8b-base. - Parameter Count: 8 billion parameters.
- Context Length: Supports an 8192-token context window.
- Training Dataset: Specialized training on
capsd_marin-8b-base-n80000-opc__mix_code_random_b4000_s0.
Intended Use
While specific intended uses and limitations are not detailed in the provided information, the fine-tuning on a dataset with "code" in its name suggests potential applicability for tasks involving code generation, analysis, or understanding. Further evaluation would be needed to confirm its specific strengths and optimal use cases.