Grogros/dmWM-meta-llama-Llama-3.2-1B-Instruct-ft-OpenMathInstruct-AlpacaGPT4-OpenWebText
Grogros/dmWM-meta-llama-Llama-3.2-1B-Instruct-ft-OpenMathInstruct-AlpacaGPT4-OpenWebText is a 1 billion parameter instruction-tuned language model, fine-tuned from meta-llama/Llama-3.2-1B-Instruct. It was trained with a context length of 32768 tokens. This model is a general-purpose instruction-following LLM, but specific differentiators or primary use cases are not detailed in its documentation.
Loading preview...
Overview
This model, dmWM-meta-llama-Llama-3.2-1B-Instruct-ft-OpenMathInstruct-AlpacaGPT4-OpenWebText, is a 1 billion parameter instruction-tuned language model developed by Grogros. It is a fine-tuned version of the meta-llama/Llama-3.2-1B-Instruct base model. The training process involved specific hyperparameters, including a learning rate of 2e-05, a total train batch size of 32 (with train_batch_size: 4 and gradient_accumulation_steps: 8), and 2500 training steps. The optimizer used was ADAFACTOR with a cosine learning rate scheduler and a warmup ratio of 0.1. The model was trained using Transformers 4.46.3 and PyTorch 2.5.1.post303.
Key Capabilities
- Instruction Following: As an instruction-tuned model, it is designed to follow user prompts and generate relevant responses.
- Base Model: Built upon the Llama-3.2-1B-Instruct architecture, inheriting its foundational language understanding capabilities.
Good for
- General-purpose text generation: Suitable for various tasks requiring instruction adherence.
- Further experimentation: Can serve as a base for additional fine-tuning on specific datasets or tasks.