FlyPig23/Llama3.2-3B_Paper_Impact_code_SFT_1ep
FlyPig23/Llama3.2-3B_Paper_Impact_code_SFT_1ep is a 3.2 billion parameter Llama 3.2-Instruct model fine-tuned by FlyPig23. This model is specifically trained on the paper_impact_code_train dataset for one epoch, demonstrating a low loss of 0.0870 on its evaluation set. It is optimized for tasks related to code generation or analysis, particularly within the context of academic paper impact.
Loading preview...
Model Overview
FlyPig23/Llama3.2-3B_Paper_Impact_code_SFT_1ep is a 3.2 billion parameter language model, fine-tuned from the meta-llama/Llama-3.2-3B-Instruct base model. This iteration has undergone one epoch of supervised fine-tuning (SFT) on the paper_impact_code_train dataset.
Key Characteristics
- Base Model:
meta-llama/Llama-3.2-3B-Instruct - Parameter Count: 3.2 billion
- Context Length: 32768 tokens
- Fine-tuning Dataset:
paper_impact_code_train - Evaluation Performance: Achieved a loss of 0.0870 on the evaluation set, indicating strong performance on the specific fine-tuning task.
Training Details
The model was trained with a learning rate of 2e-05, a total batch size of 128 (achieved with train_batch_size 8 and gradient_accumulation_steps 4), and utilized a cosine learning rate scheduler with a 0.1 warmup ratio. The training was conducted for 1.0 epoch across 4 devices using a multi-GPU distributed setup.
Potential Use Cases
Given its fine-tuning on a code-related dataset, this model is likely suitable for:
- Code generation tasks, especially those related to academic research or paper analysis.
- Understanding or summarizing code snippets within the context of research papers.
- Assisting with tasks that bridge natural language descriptions of research impact with corresponding code implementations.