AmberYifan/capsd-opc-dedup-marin-8b-base-code_ppl_b1000_s0
AmberYifan/capsd-opc-dedup-marin-8b-base-code_ppl_b1000_s0 is an 8 billion parameter language model, fine-tuned from marin-community/marin-8b-base. This model is specifically trained on a deduplicated dataset with a focus on code-related perplexity optimization. It is designed for tasks benefiting from improved code understanding and generation capabilities.
Loading preview...
Model Overview
This model, AmberYifan/capsd-opc-dedup-marin-8b-base-code_ppl_b1000_s0, is an 8 billion parameter language model derived from the marin-community/marin-8b-base architecture. It has undergone fine-tuning on a specialized dataset, capsd_marin-8b-base-n80000-opc-dedup80k__mix_code_ppl_b1000_s0, which emphasizes code-related content and perplexity optimization.
Key Characteristics
- Base Model: Fine-tuned from
marin-community/marin-8b-base. - Parameter Count: 8 billion parameters.
- Training Focus: Optimized for code-related tasks through specific dataset fine-tuning.
- Context Length: Supports an 8192-token context window.
Training Details
The model was trained with a learning rate of 1e-05, using a cosine LR scheduler and AdamW optimizer. The training involved 1 epoch with a total batch size of 64 across 4 GPUs. The training procedure utilized Transformers 5.7.0 and Pytorch 2.13.0+cu130.
Potential Use Cases
While specific intended uses and limitations require further information, the model's fine-tuning on a code-centric dataset suggests potential applications in:
- Code generation and completion.
- Code understanding and analysis.
- Tasks requiring strong performance on programming language data.