mlfoundations-dev/oh-dcft-v1.2_no-curation_gpt-4o-mini_wo_airoboros
The mlfoundations-dev/oh-dcft-v1.2_no-curation_gpt-4o-mini_wo_airoboros model is an 8 billion parameter language model, fine-tuned from Meta-Llama-3.1-8B. It was trained on the mlfoundations-dev/oh-dcft-v1.2_no-curation_gpt-4o-mini_wo_airoboros dataset, achieving a validation loss of 0.6514. This model is a general-purpose fine-tune, suitable for a wide range of natural language processing tasks.
Loading preview...
Model Overview
This model, oh-dcft-v1.2_no-curation_gpt-4o-mini_wo_airoboros, is an 8 billion parameter language model derived from Meta-Llama-3.1-8B. It has been fine-tuned using a specific dataset, mlfoundations-dev/oh-dcft-v1.2_no-curation_gpt-4o-mini_wo_airoboros, to adapt its capabilities.
Training Details
The model underwent 3 epochs of training with a learning rate of 5e-06 and a total batch size of 512 across 32 GPUs. The training process utilized an Adam optimizer and a constant learning rate scheduler with a warmup ratio of 0.1. Key training results include:
- Final Validation Loss: 0.6514
Technical Specifications
- Base Model: meta-llama/Meta-Llama-3.1-8B
- Parameters: 8 Billion
- Context Length: 32768 tokens
- Frameworks: Transformers 4.44.2, Pytorch 2.4.0, Datasets 3.0.2, Tokenizers 0.19.1
Intended Use
While specific intended uses and limitations are not detailed in the provided information, as a fine-tuned Llama 3.1 variant, it is generally suitable for various generative AI and natural language understanding tasks.