Model Overview
FlyPig23/Llama3.2-3B_Paper_Impact_code_SFT_1ep is a 3.2 billion parameter language model, fine-tuned from the meta-llama/Llama-3.2-3B-Instruct base model. This iteration has undergone one epoch of supervised fine-tuning (SFT) on the paper_impact_code_train dataset.
Key Characteristics
- Base Model:
meta-llama/Llama-3.2-3B-Instruct - Parameter Count: 3.2 billion
- Context Length: 32768 tokens
- Fine-tuning Dataset:
paper_impact_code_train - Evaluation Performance: Achieved a loss of 0.0870 on the evaluation set, indicating strong performance on the specific fine-tuning task.
Training Details
The model was trained with a learning rate of 2e-05, a total batch size of 128 (achieved with train_batch_size 8 and gradient_accumulation_steps 4), and utilized a cosine learning rate scheduler with a 0.1 warmup ratio. The training was conducted for 1.0 epoch across 4 devices using a multi-GPU distributed setup.
Potential Use Cases
Given its fine-tuning on a code-related dataset, this model is likely suitable for:
- Code generation tasks, especially those related to academic research or paper analysis.
- Understanding or summarizing code snippets within the context of research papers.
- Assisting with tasks that bridge natural language descriptions of research impact with corresponding code implementations.