Gueule-d-ange/llama32-3b-redo-sft
Gueule-d-ange/llama32-3b-redo-sft is a 3.2 billion parameter language model fine-tuned from the Meta Llama-3.2-3B architecture. This model has been specifically fine-tuned on the 'sft_mix' dataset, indicating an optimization for supervised fine-tuning tasks. It is designed for general language generation and understanding within its 32768 token context window, offering a compact yet capable solution for various NLP applications.
Loading preview...
Model Overview
Gueule-d-ange/llama32-3b-redo-sft is a 3.2 billion parameter language model, derived from the Meta Llama-3.2-3B base architecture. It has undergone supervised fine-tuning (SFT) using the sft_mix dataset, aiming to enhance its performance on instruction-following and general conversational tasks.
Key Characteristics
- Base Model: Fine-tuned from
meta-llama/Llama-3.2-3B. - Parameter Count: 3.2 billion parameters, offering a balance between performance and computational efficiency.
- Context Length: Supports a context window of 32768 tokens, allowing for processing longer inputs and generating more coherent, extended outputs.
- Training Objective: Optimized through supervised fine-tuning on the
sft_mixdataset, suggesting improved adherence to instructions and better conversational abilities.
Training Details
The model was trained for 3 epochs with a learning rate of 2e-05 and a total batch size of 64 (achieved with train_batch_size: 8 and gradient_accumulation_steps: 8). The training utilized the AdamW optimizer with cosine learning rate scheduling and a warmup ratio of 0.1. The final validation loss achieved was 0.6408.
Intended Uses
This model is suitable for applications requiring a compact yet capable language model for tasks such as:
- General text generation.
- Instruction following.
- Conversational AI.
- Text summarization or completion where a 3.2B parameter model is sufficient.