iamaber/mistral-7b-pubmedqa-lora-plus

TEXT GENERATIONConcurrency Cost:1Model Size:7BQuant:FP8Ctx Length:4kPublished:Apr 5, 2026License:apache-2.0Architecture:Transformer Open Weights Cold

The iamaber/mistral-7b-pubmedqa-lora-plus model is a 7 billion parameter Mistral-7B-Instruct-v0.3 variant, fine-tuned using LoRA+ on the PubMedQA dataset. This model is specifically optimized for medical question answering tasks, demonstrating a PubMedQA accuracy of 0.4500. It is designed for applications requiring specialized knowledge in the biomedical domain, particularly for answering 'yes', 'no', or 'maybe' questions based on medical literature.

Loading preview...

Model Overview

The iamaber/mistral-7b-pubmedqa-lora-plus is a specialized language model built upon the mistralai/Mistral-7B-Instruct-v0.3 architecture. It has been fine-tuned using the LoRA+ method on the qiaojin/PubMedQA dataset, specifically the pqa_labeled subset. This targeted training aims to enhance its performance in medical question-answering scenarios.

Key Capabilities and Performance

  • Medical Question Answering: The model is designed to answer questions within the biomedical domain, particularly those requiring a 'yes', 'no', or 'maybe' response, as evidenced by its training on PubMedQA.
  • PubMedQA Accuracy: Achieves an accuracy of 0.4500 on the PubMedQA evaluation set, with a macro F1 score of 0.2069 and a weighted F1 score of 0.2793.
  • Training Details: Fine-tuned over 3 epochs with 900 training examples and 100 evaluation examples, using a learning rate of 5e-05.

Limitations

  • Medical MMLU Performance: The model shows limited performance on broader medical knowledge, with a Medical MMLU accuracy of 0.1600 across 50 samples. Specific subjects like clinical knowledge, college medicine, medical genetics, professional medicine, and virology showed 0.0000 accuracy in the provided breakdown.
  • Confusion Matrix Insights: The PubMedQA confusion matrix indicates a strong bias towards predicting 'yes', with 45 correct 'yes' predictions but 40 'no' and 15 'maybe' questions incorrectly classified as 'yes'. This suggests a need for careful interpretation of its 'no' and 'maybe' responses.

Ideal Use Cases

This model is best suited for:

  • Preliminary Medical Information Retrieval: Answering straightforward, fact-based questions from medical texts where a 'yes/no/maybe' answer is expected.
  • Biomedical Research Support: Assisting researchers in quickly sifting through medical literature for specific answers.
  • Specialized QA Systems: Integration into applications focused solely on PubMedQA-style medical inquiries.

Finetunes
1 models