Grogros/dmWM-llama-3.2-1B-Instruct-kgw_wmtoken-OWT-3WT-DistillationWM-Al4-WT3-d4-v1
The Grogros/dmWM-llama-3.2-1B-Instruct-kgw_wmtoken-OWT-3WT-DistillationWM-Al4-WT3-d4-v1 is a 1 billion parameter instruction-tuned causal language model, fine-tuned from the Meta Llama 3.2-1B-Instruct architecture. This model features a 32768 token context length and was trained with specific hyperparameters including a learning rate of 2e-05 and 10000 training steps. Its primary differentiator and specific use cases are not detailed in the provided information, suggesting it may be a base model for further specialization or research.
Loading preview...
Overview
This model, dmWM-llama-3.2-1B-Instruct-kgw_wmtoken-OWT-3WT-DistillationWM-Al4-WT3-d4-v1, is a 1 billion parameter instruction-tuned language model. It is built upon the meta-llama/Llama-3.2-1B-Instruct architecture, indicating its foundation in the Llama family of models. The model supports a substantial context length of 32768 tokens, which is beneficial for processing longer inputs and generating more coherent, extended responses.
Training Details
The model underwent a fine-tuning process with specific hyperparameters. Key training parameters include a learning rate of 2e-05, a train_batch_size and eval_batch_size of 8, and a gradient_accumulation_steps of 8, resulting in a total_train_batch_size of 64. An ADAFACTOR optimizer was utilized, and the training comprised 10000 steps with a cosine learning rate scheduler and a 0.1 warmup ratio. The training was conducted using Transformers 4.46.3, Pytorch 2.5.1.post303, Datasets 3.2.0, and Tokenizers 0.20.3.
Key Characteristics
- Base Model: Fine-tuned from Meta Llama 3.2-1B-Instruct.
- Parameter Count: 1 billion parameters.
- Context Length: 32768 tokens.
- Training Steps: 10000 steps with specific optimization settings.
Good for
Given the available information, this model is suitable for researchers and developers looking for a 1 billion parameter Llama-based instruction-tuned model with a large context window for experimentation. Its specific strengths or intended applications are not detailed, suggesting it could serve as a versatile base for further fine-tuning on specialized datasets or tasks where a compact yet capable model with extended context handling is required.