Model Overview

StanfordAIMI/GREEN-RadPhi2 is a 3 billion parameter language model, fine-tuned from the existing StanfordAIMI/RadPhi-2 architecture. The model was trained over 12 epochs, achieving a final validation loss of 0.0816.

Training Details

The training process utilized specific hyperparameters to optimize performance:

• Learning Rate: 0.0001
• Batch Size: 8 (train and eval), with a total effective train batch size of 2048 due to gradient accumulation
• Optimizer: Adam with betas=(0.9, 0.95) and epsilon=1e-08
• Scheduler: Cosine learning rate scheduler with a 0.05 warmup ratio

Performance

During training, the model consistently reduced its validation loss, reaching a minimum of 0.0786 at epoch 8.95 and concluding with 0.0816. This indicates a well-converged fine-tuning process.

Limitations

The specific dataset used for fine-tuning is not detailed in the model card, and further information regarding intended uses and limitations is currently unavailable. Users should exercise caution and conduct thorough evaluations for specific applications.