Grogros/dmWM-llama-3.2-1B-Instruct-KGWB-OWT_WMBoundary-OWT-WB-v3
The Grogros/dmWM-llama-3.2-1B-Instruct-KGWB-OWT_WMBoundary-OWT-WB-v3 is a 1 billion parameter instruction-tuned causal language model, fine-tuned by Grogros from the meta-llama/Llama-3.2-1B-Instruct base model. It was specifically fine-tuned on the OpenWebText dataset, suggesting a focus on broad text generation and understanding from web-scale data. With a context length of 32768 tokens, this model is designed for general-purpose conversational AI and text-based tasks requiring a substantial context window.
Loading preview...
Overview
This model, dmWM-llama-3.2-1B-Instruct-KGWB-OWT_WMBoundary-OWT-WB-v3, is a 1 billion parameter instruction-tuned language model developed by Grogros. It is a fine-tuned variant of the meta-llama/Llama-3.2-1B-Instruct base model, specifically enhanced through training on the OpenWebText dataset.
Key Capabilities
- Instruction Following: Inherits instruction-following capabilities from its Llama-3.2-1B-Instruct base.
- Broad Text Understanding: Benefits from fine-tuning on the OpenWebText dataset, providing a wide range of general knowledge and text generation abilities.
- Large Context Window: Supports a context length of 32768 tokens, enabling it to process and generate longer sequences of text.
Training Details
The model was trained with the following key hyperparameters:
- Learning Rate: 2e-05
- Batch Size: 8 (train and eval), with a total effective batch size of 64 due to gradient accumulation steps.
- Optimizer: ADAFACTOR
- Scheduler: Cosine learning rate scheduler with a 0.1 warmup ratio.
- Training Steps: 5000 steps.
Good For
- General-purpose text generation and understanding tasks.
- Applications requiring a model with a substantial context window for processing longer inputs.
- Use cases where a compact, instruction-tuned model with broad web-based knowledge is beneficial.