AmberYifan/capsdnum-marin-8b-base-code_random_b1000_s0
The AmberYifan/capsdnum-marin-8b-base-code_random_b1000_s0 model is an 8 billion parameter language model, fine-tuned from marin-community/marin-8b-base. This model has been specifically adapted using the capsd_marin-8b-base-n80000-opc__mix_code_random_b1000_s0 dataset, suggesting an optimization for code-related tasks. It operates with a context length of 8192 tokens, making it suitable for processing moderately long code sequences or technical documentation.
Loading preview...
Model Overview
AmberYifan/capsdnum-marin-8b-base-code_random_b1000_s0 is an 8 billion parameter language model, fine-tuned from the marin-community/marin-8b-base architecture. This model has been specialized through training on the capsd_marin-8b-base-n80000-opc__mix_code_random_b1000_s0 dataset, indicating a focus on code-related applications.
Key Characteristics
- Base Model: Fine-tuned from
marin-community/marin-8b-base. - Parameter Count: 8 billion parameters.
- Context Length: Supports an 8192-token context window.
- Training Data: Specialized on a mixed code dataset, suggesting enhanced performance for programming tasks.
Training Details
The model was trained using the following hyperparameters:
- Learning Rate: 1e-05
- Batch Size: A total training batch size of 64 (with 2 per device and 8 gradient accumulation steps) and an evaluation batch size of 32.
- Optimizer: ADAMW_TORCH with default betas and epsilon.
- Scheduler: Cosine learning rate scheduler with 0.03 warmup steps.
- Epochs: Trained for 1 epoch.
Potential Use Cases
Given its fine-tuning on a code-centric dataset, this model is likely well-suited for:
- Code generation and completion.
- Code analysis and understanding.
- Assisting with programming tasks where a moderate context window is beneficial.