AmberYifan/capsd-less-ultra-humaneval-opc-marin-8b-base-code_less_b8000_s0

Model Overview

This model, AmberYifan/capsd-less-ultra-humaneval-opc-marin-8b-base-code_less_b8000_s0, is an 8 billion parameter language model. It is a fine-tuned iteration of the marin-community/marin-8b-base architecture, indicating a foundation in a robust base model. The fine-tuning process involved the capsd_marin-8b-base-n80000-opc__mix_code_less_b8000_s0 dataset, suggesting specialized training for particular tasks or data distributions.

Training Details

The model was trained using a learning rate of 1e-05, with a train_batch_size of 2 and eval_batch_size of 8 across 4 GPUs. It utilized an AdamW optimizer and a cosine learning rate scheduler with 0.03 warmup steps over 1 epoch. The training was conducted using Transformers 5.7.0, Pytorch 2.13.0+cu130, Datasets 4.0.0, and Tokenizers 0.22.2.

Key Characteristics

• Parameter Count: 8 billion parameters.
• Context Length: Supports a context length of 8192 tokens.
• Base Model: Fine-tuned from marin-community/marin-8b-base.

Intended Use Cases

While specific intended uses are not detailed, its foundational nature and fine-tuning suggest applicability in general language tasks, potentially with an emphasis on areas covered by its training dataset. Developers should consider its 8B parameter size and 8192 token context window for applications requiring moderate complexity and context.