barc0/Llama-3.1-ARC-Potpourri-Transduction-8B

Model Overview

This model, barc0/Llama-3.1-ARC-Potpourri-Transduction-8B, is an 8 billion parameter instruction-tuned language model based on meta-llama/Meta-Llama-3.1-8B-Instruct. It has undergone extensive fine-tuning across several specialized datasets, primarily focusing on 'transduction' tasks.

Key Capabilities

• Specialized Transduction: Fine-tuned on datasets like barc0/transduction_heavy_100k_jsonl, barc0/transduction_heavy_suggestfunction_100k_jsonl, barc0/transduction_rearc_dataset_400k, and augmented problems generated by GPT-4 and GPT-4o-mini.
• Problem Generation: Training data includes problems generated by advanced LLMs, suggesting a capability in understanding and formulating complex tasks.
• Optimized Performance: Achieved a final validation loss of 0.0219 after 3 epochs of training, indicating strong performance on its specialized datasets.

Training Details

The model was trained with a learning rate of 1e-05, a total batch size of 128, and utilized 8 GPUs. The training process involved 3 epochs, with a cosine learning rate scheduler and a warmup ratio of 0.1.