Model Overview
The barc0/cot-transduction-only-arc is an 8 billion parameter instruction-tuned model, built upon the meta-llama/Meta-Llama-3.1-8B-Instruct architecture. It has been fine-tuned using several specialized datasets, specifically barc0/trans_only_cot_100k-gpt4omini-description, barc0/trans_only_cot_100k-gpt4-description, and barc0/trans_only_cot_200k_HEAVY_gpt4o-description.
Key Characteristics
- Base Model: Fine-tuned from Meta-Llama-3.1-8B-Instruct.
- Parameter Count: 8 billion parameters.
- Context Length: Supports a context length of 32768 tokens.
- Training Focus: Specialized training on datasets designed for 'transduction only' chain-of-thought (CoT) tasks.
- Performance: Achieved a validation loss of 0.0253 during training, indicating strong performance on its specific fine-tuning objective.
Training Details
The model was trained for 3 epochs with a learning rate of 1e-05, using a total batch size of 128 across 8 GPUs. The optimizer used was Adam with betas=(0.9, 0.999) and epsilon=1e-08, employing a cosine learning rate scheduler with a 0.1 warmup ratio.
Potential Use Cases
This model is likely suitable for applications requiring specialized chain-of-thought transduction capabilities, particularly those aligned with the data it was fine-tuned on. Developers should consider its specific training focus when evaluating its fit for their particular use cases.