ddrkane/kore
ddrkane/kore is a 12 billion parameter language model trained from scratch by ddrkane. With a context length of 32768 tokens, this model was developed using a cosine learning rate scheduler and AdamW optimizer. Due to limited information, its specific primary use case and unique differentiators are not detailed.
Loading preview...
Overview
ddrkane/kore is a 12 billion parameter language model developed by ddrkane. This model was trained from scratch, though specific details about the dataset are not provided in the available documentation. It supports a substantial context length of 32768 tokens.
Training Details
The training process utilized a learning rate of 0.0001, a train batch size of 2, and an evaluation batch size of 1, with a seed of 42. Gradient accumulation steps were set to 8, resulting in a total effective training batch size of 16. The optimizer used was AdamW with default betas and epsilon, and a cosine learning rate scheduler with a warmup ratio of 0.05 was applied over 2 epochs. The model was developed using Transformers 4.51.3, PyTorch 2.4.1, Datasets 3.6.0, and Tokenizers 0.21.4.
Key Capabilities
- Large Context Window: Supports processing up to 32768 tokens, enabling handling of extensive inputs.
- From-Scratch Training: Indicates potential for unique characteristics not inherited from existing base models.
Good for
- Exploratory Research: Suitable for researchers interested in models trained from scratch with specific hyperparameters.
- Applications requiring long context: Its large context window makes it suitable for tasks involving extensive text analysis or generation.