Harryxun/llama-pretrained
The Harryxun/llama-pretrained model is a 1 billion parameter language model, fine-tuned from Meta Llama-3.2-1B. This model was trained using specific hyperparameters including a learning rate of 0.0005 and cosine learning rate scheduler over 2 epochs. While its specific training dataset and intended uses are not detailed, it represents a base model for further fine-tuning or research into Llama-3.2 architecture at a smaller scale.
Loading preview...
Model Overview
The Harryxun/llama-pretrained model is a 1 billion parameter language model, derived from the meta-llama/Llama-3.2-1B architecture. This model has undergone a fine-tuning process, though the specific dataset used for this training is not disclosed in the available documentation.
Training Details
The fine-tuning procedure involved several key hyperparameters:
- Learning Rate: 0.0005
- Batch Sizes:
train_batch_sizeof 32,eval_batch_sizeof 32, withgradient_accumulation_stepsset to 8, resulting in atotal_train_batch_sizeof 256. - Optimizer:
adamw_torch_fusedwith default betas and epsilon. - Scheduler: Cosine learning rate scheduler with 1000 warmup steps.
- Epochs: Trained for 2 epochs.
- Precision: Utilized Native AMP for mixed-precision training.
Current Status and Limitations
As of the current documentation, further information regarding the model's specific intended uses, limitations, and the dataset it was trained on is needed. This model serves as a fine-tuned base, potentially suitable for researchers or developers looking to experiment with the Llama-3.2 architecture at a 1 billion parameter scale, with the understanding that its specific capabilities and performance characteristics are yet to be fully detailed.