Nelis5174473/GovLLM-7B-ultra
Nelis5174473/GovLLM-7B-ultra is a 7 billion parameter question-answering model specifically fine-tuned for information about the Dutch Government. Based on the Mistral architecture and fine-tuned from BramVanroy/GEITje-7B-ULTRA, this model excels at answering Dutch-language queries related to government topics. It was trained using SFT and LoRA on a dedicated Dutch question-answer dataset, making it highly specialized for this domain.
Loading preview...
GovLLM-7B-ultra: Specialized Dutch Government QA Model
GovLLM-7B-ultra is a 7 billion parameter language model designed for question answering about the Dutch Government. It is a fine-tuned version of the Dutch conversational model BramVanroy/GEITje-7B-ULTRA, which itself is based on the Mistral architecture. The model was developed by Nelis5174473.
Key Capabilities
- Specialized Question Answering: Optimized to provide answers to questions specifically concerning the Dutch Government.
- Dutch Language Support: Built upon a Dutch conversational model and trained on a Dutch question-answer dataset, ensuring high proficiency in the Dutch language.
- Efficient Fine-tuning: Utilizes Supervised Fine-Tuning (SFT) and LoRA (Low-Rank Adaptation) for efficient adaptation to its specific domain.
Training Details
The model was fine-tuned for 3 epochs on a Dutch question-answer pair dataset derived from the Dutch Government. Training was conducted on an Nvidia A100 (40GB VRAM) and took approximately 2 hours. Key hyperparameters included a model_max_length of 2048, a learning rate of 0.00003, and int8 quantization with PEFT enabled.
Good For
- Applications requiring precise question answering on topics related to the Dutch Government.
- Building chatbots or information retrieval systems focused on Dutch public administration.