YuvrajSingh9886/medicinal-QnA-phi2-custom
YuvrajSingh9886/medicinal-QnA-phi2-custom is a 3 billion parameter Phi-2 model by Microsoft, fine-tuned by YuvrajSingh9886 for medicinal question-and-answer tasks. This model is specifically optimized for generating responses relevant to health counseling conversations, leveraging the Amod/mental_health_counseling_conversations dataset. It is designed to assist in the development of health counseling chatbots, providing specialized conversational capabilities within a compact 3B parameter footprint and a 2048 token context length.
Loading preview...
Model Overview
YuvrajSingh9886/medicinal-QnA-phi2-custom is a 3 billion parameter language model, fine-tuned from Microsoft's Phi-2 architecture. The model was developed by YuvrajSingh9886 as a proof-of-concept for specialized conversational AI in the medical domain.
Key Capabilities
- Specialized Q&A: Fine-tuned on the Amod/mental_health_counseling_conversations dataset, enabling it to generate responses for medicinal and mental health-related queries.
- Health Counseling Chatbot Development: Primarily intended for use in creating health counseling chatbots, with a focus on generating relevant and contextually appropriate answers.
- Quantization Support: Utilizes BitsandBytes for 4-bit precision loading (NF4 quantization with double quantization), making it efficient for deployment on hardware with limited VRAM.
Training Details
The model was trained for 10 epochs on a single RTX 4090 (24GB VRAM) using QLoRA for quantization. Key hyperparameters include a learning rate of 2e-4, a batch size of 6, and a LoRA attention dimension of 64. The training achieved a final loss of 2.229 and a validation loss of 2.223.
Intended Use Cases
This model is suitable for:
- Prototyping Health Counseling Chatbots: Developers can use this model as a base for building conversational agents focused on mental health and general medicinal Q&A.
- Research in Specialized LLMs: Exploring the effectiveness of fine-tuning smaller models like Phi-2 for niche applications.
Limitations
It is important to note that this model is a hobby project and a proof-of-concept. It is not intended for production use without thorough evaluation and consideration of its implications, especially in sensitive domains like health counseling.