michelinolinolino/gemma4-4b-sci
michelinolinolino/gemma4-4b-sci is a 7.9 billion parameter scientific-domain fine-tune of Gemma 4 E4B, developed by Michele Banfi. This model is specifically optimized for scientific question answering and text generation, trained on 30,000 examples from OpenSciLM/OS_Train_Data and SciRIFF using QLoRA. It demonstrates strong performance in scientific correctness, matching or exceeding larger models like OpenScholar-8B on tasks like SciFact and PubMedQA.
Loading preview...
Overview
michelinolinolino/gemma4-4b-sci is an experimental scientific-domain fine-tune of the Gemma 4 E4B base model, developed by Michele Banfi. It utilizes QLoRA (4-bit) and Supervised Fine-Tuning (SFT) via Unsloth, focusing on language layers while freezing the vision encoder. The model was trained for one epoch on 30,000 examples from the OpenSciLM/OS_Train_Data and SciRIFF datasets.
Key Capabilities & Performance
This model is designed for generation-only tasks within the scientific domain. Despite being an early-stage research experiment, it shows promising results:
- Scientific Question Answering: Achieves 77.9% accuracy on SciFact and 81.5% accuracy on PubMedQA, matching or exceeding the performance of OpenScholar-8B (a model with twice the parameters) in terms of correctness.
- Domain Specialization: Fine-tuned specifically on scientific literature, making it suitable for tasks requiring deep scientific knowledge.
Limitations
As an early-stage experiment, users should expect hallucinations and factual errors. The model currently lacks a retrieval pipeline, which impacts its citation F1 scores compared to models like OpenScholar-8B that incorporate retrieval.
Use Cases
This model is particularly well-suited for:
- Scientific text generation: Generating explanations or summaries of scientific concepts.
- Research in scientific NLP: Exploring fine-tuning approaches for domain-specific language models.
- Question answering in scientific contexts: Answering factual questions based on scientific literature, where correctness is prioritized over citation accuracy.