cuong1692001/Terminal-16k-top80
Terminal-16k-top80 is an 8 billion parameter language model developed by cuong1692001, fine-tuned from Terminal_complete_8k. This model is specifically trained on the nemotron_complete_top_80_16k dataset, indicating an optimization for tasks related to that specific data distribution. It is designed for applications requiring a model with a 32768 token context length, building upon its base model's capabilities.
Loading preview...
Model Overview
Terminal-16k-top80 is an 8 billion parameter language model, fine-tuned by cuong1692001. It is a specialized iteration of the cuong1692001/Terminal_complete_8k base model, with a notable context length of 32768 tokens.
Key Characteristics
- Base Model: Fine-tuned from Terminal_complete_8k.
- Training Data: Specifically trained on the
nemotron_complete_top_80_16kdataset, suggesting a focus on tasks relevant to this data distribution. - Context Length: Supports a substantial context window of 32768 tokens.
Training Details
The model was trained using the following hyperparameters:
- Learning Rate: 1e-05
- Optimizer: ADAMW_TORCH with default betas and epsilon.
- Epochs: 2.0
- Batch Size: A
train_batch_sizeof 1 andeval_batch_sizeof 8 were used, with a total distributed training batch size of 4 across 4 GPUs.
Intended Use
While specific intended uses and limitations are not detailed in the provided information, its fine-tuning on a particular dataset and large context window suggest suitability for applications requiring deep contextual understanding within that domain.