cuong1692001/Terminal-16k-top50
The cuong1692001/Terminal-16k-top50 is an 8 billion parameter language model, fine-tuned from Terminal-complete_8k. This model was specifically trained on the nemotron_complete_top_50_16k dataset, suggesting an optimization for tasks related to its training data. With a 32768 token context length, it is suitable for applications requiring processing of longer sequences.
Loading preview...
Model Overview
The cuong1692001/Terminal-16k-top50 is an 8 billion parameter language model, derived from the Terminal-complete_8k base model. It has been fine-tuned with a focus on the nemotron_complete_top_50_16k dataset, indicating a specialization towards the characteristics and content of this particular data.
Key Characteristics
- Base Model: Fine-tuned from
Terminal-complete_8k. - Parameter Count: 8 billion parameters.
- Context Length: Supports a context window of 32768 tokens, enabling the processing of extensive inputs and generating longer outputs.
- Training Data: Specialized training on the
nemotron_complete_top_50_16kdataset.
Training Details
The model underwent training for 2 epochs using a learning rate of 1e-05 and an AdamW optimizer. The training utilized a distributed setup across 4 GPUs with a total batch size of 4, employing a cosine learning rate scheduler.
Potential Use Cases
Given its fine-tuning on a specific dataset and extended context length, this model is likely suitable for applications that align with the nature of the nemotron_complete_top_50_16k data, particularly those requiring detailed understanding or generation over long text passages.