AmberYifan/capmix-marin-8b-base-uniform
AmberYifan/capmix-marin-8b-base-uniform is an 8 billion parameter causal language model, fine-tuned from the marin-community/marin-8b-base architecture. This model is trained on the capmix_marin-8b-base__mix_uniform dataset, offering a base model for further specialization. With an 8192 token context length, it provides a solid foundation for various natural language processing tasks.
Loading preview...
Model Overview
AmberYifan/capmix-marin-8b-base-uniform is an 8 billion parameter language model, serving as a fine-tuned version of the marin-community/marin-8b-base architecture. It was specifically trained on the capmix_marin-8b-base__mix_uniform dataset, indicating a specialized training regimen for particular data distributions or tasks. The model supports a context length of 8192 tokens, allowing it to process and generate longer sequences of text.
Training Details
The model underwent training with the following key hyperparameters:
- Learning Rate: 1e-05
- Batch Sizes:
train_batch_sizeof 4,eval_batch_sizeof 8 - Optimizer: ADAMW_TORCH with default betas and epsilon
- LR Scheduler: Cosine type with 0.03 warmup steps
- Epochs: 1
Training was conducted using a multi-GPU setup with 4 devices, a gradient_accumulation_steps of 4, resulting in a total_train_batch_size of 64 and total_eval_batch_size of 32.
Framework Versions
The training environment utilized:
- Transformers 5.7.0
- Pytorch 2.13.0+cu130
- Datasets 4.0.0
- Tokenizers 0.22.2
Intended Use
While specific intended uses and limitations are not detailed, this model, being a base fine-tuned version, is suitable for further adaptation or as a foundational component in applications requiring an 8B parameter model with a substantial context window.