AmberYifan/capmix-marin-8b-base-loss
AmberYifan/capmix-marin-8b-base-loss is an 8 billion parameter language model, fine-tuned from marin-community/marin-8b-base. This model was trained on the capmix_marin-8b-base__mix_loss dataset, indicating a specialized training objective. It is designed for tasks aligned with its specific fine-tuning data, offering capabilities derived from its base model and subsequent optimization.
Loading preview...
Overview
AmberYifan/capmix-marin-8b-base-loss is an 8 billion parameter language model, fine-tuned from the marin-community/marin-8b-base architecture. This model underwent a specific fine-tuning process using the capmix_marin-8b-base__mix_loss dataset.
Training Details
The model was trained with a learning rate of 1e-05, a train_batch_size of 4, and an eval_batch_size of 8. It utilized a multi-GPU setup with 4 devices and a gradient_accumulation_steps of 4, resulting in a total_train_batch_size of 64. The optimizer used was ADAMW_TORCH with standard betas and epsilon, and a cosine learning rate scheduler with 0.03 warmup steps. Training was conducted for 1 epoch.
Key Characteristics
- Base Model: Fine-tuned from
marin-community/marin-8b-base. - Parameter Count: 8 billion parameters.
- Context Length: Supports an 8192 token context window.
- Training Dataset: Specialized fine-tuning on
capmix_marin-8b-base__mix_loss.
Intended Use
Given its fine-tuning on a specific dataset, this model is likely best suited for tasks and applications that align with the characteristics and domain of the capmix_marin-8b-base__mix_loss dataset. Developers should consider the nature of this dataset when evaluating its suitability for their particular use cases.