Heralax/army-pretrain-1
Heralax/army-pretrain-1 is a 7 billion parameter pretrained base model, serving as the foundation for the Mistrillitary model. This model is designed as a foundational language model, providing a robust base for further fine-tuning and specialized applications. Its primary utility lies in serving as a strong starting point for developing more specific AI models, particularly within the Mistrillitary ecosystem. The model was trained with a learning rate of 2e-05 and a cosine learning rate scheduler over 11 epochs.
Loading preview...
Model Overview
Heralax/army-pretrain-1 is a 7 billion parameter pretrained base model developed by Heralax. It serves as the foundational layer for the Mistrillitary model, providing a robust starting point for various natural language processing tasks. This model is designed to be a general-purpose language understanding and generation base, intended for further specialization through fine-tuning.
Training Details
The model was trained using specific hyperparameters to optimize its performance:
- Learning Rate: 2e-05
- Batch Size: A
train_batch_sizeof 2 andeval_batch_sizeof 1 were used, with atotal_train_batch_sizeof 72 across 6 multi-GPU devices. - Optimizer: Adam with betas=(0.9, 0.999) and epsilon=1e-08.
- Scheduler: A cosine learning rate scheduler with 136 warmup steps.
- Epochs: The training process ran for 11 epochs.
Frameworks Used
The development leveraged several key machine learning frameworks:
- Transformers 4.45.0.dev0
- Pytorch 2.3.1+cu121
- Datasets 2.21.0
- Tokenizers 0.19.1
Intended Use
This model is best suited as a base model for developers looking to build custom language models. It provides a strong, pretrained foundation that can be adapted to specific domains or tasks through additional fine-tuning, particularly within the Mistrillitary model family.