AmberYifan/capsd-medcase-marin-8b-base-medicine_random_b8000_s0

Model Overview

This model, marin-8b-base_medicine_random_b8000_s0, is an 8 billion parameter language model developed by AmberYifan. It is a fine-tuned variant of the marin-community/marin-8b-base architecture, specifically adapted for medical applications.

Key Characteristics

• Base Model: Fine-tuned from marin-community/marin-8b-base.
• Parameter Count: 8 billion parameters.
• Context Length: Supports a context window of 8192 tokens.
• Specialization: Trained on a medical-specific dataset, capsd_marin-8b-base-n13092-medicine-medcase__mix_medicine_random_b8000_s0, suggesting a focus on medical text processing and understanding.

Training Details

The model underwent training with the following hyperparameters:

• Learning Rate: 1e-05
• Batch Size: A train_batch_size of 2 and eval_batch_size of 8, with a total_train_batch_size of 64 (due to 4 devices and 8 gradient accumulation steps).
• Optimizer: ADAMW_TORCH with default betas and epsilon.
• Scheduler: Cosine learning rate scheduler with 0.03 warmup steps.
• Epochs: Trained for 1 epoch.

Intended Use

Given its fine-tuning on a medical dataset, this model is likely intended for tasks within the medical domain, such as medical text analysis, information extraction from clinical notes, or answering medical queries. Its specialization differentiates it from general-purpose LLMs by potentially offering higher accuracy and relevance for medical-specific language.