FlyPig23/Llama3.2-3B_Paper_Impact_award_SFT_1ep
FlyPig23/Llama3.2-3B_Paper_Impact_award_SFT_1ep is a 3.2 billion parameter language model, fine-tuned from Meta's Llama-3.2-3B-Instruct. This model is specifically trained on the paper_impact_award_train dataset, demonstrating a low evaluation loss of 0.0734. It is optimized for tasks related to the dataset it was fine-tuned on, offering specialized performance in that domain. The model has a context length of 32768 tokens.
Loading preview...
Model Overview
FlyPig23/Llama3.2-3B_Paper_Impact_award_SFT_1ep is a specialized language model derived from the Meta Llama-3.2-3B-Instruct architecture. This 3.2 billion parameter model has been fine-tuned for a single epoch on the paper_impact_award_train dataset, achieving an evaluation loss of 0.0734. It leverages a substantial context length of 32768 tokens, making it suitable for processing longer inputs relevant to its training data.
Key Training Details
- Base Model: meta-llama/Llama-3.2-3B-Instruct
- Fine-tuning Dataset:
paper_impact_award_train - Epochs: 1.0
- Learning Rate: 2e-05
- Batch Size: 8 (train and eval), with a total effective train batch size of 128 due to gradient accumulation.
- Optimizer: AdamW with cosine learning rate scheduler and 0.1 warmup ratio.
Potential Use Cases
Given its specific fine-tuning, this model is likely best suited for tasks closely aligned with the paper_impact_award_train dataset. Developers should consider its specialized training for applications requiring nuanced understanding or generation within that particular domain.