Fietje 2 Instruct: An Efficient Dutch LLM

Fietje 2 Instruct is an instruction-tuned variant of the Fietje base model, developed by Bram Vanroy. This model is an adaptation of Microsoft's Phi-2, specifically optimized for the Dutch language through training on 28 billion Dutch tokens. With 2.7 billion parameters, it offers an efficient solution for Dutch text generation, demonstrating performance comparable to larger Dutch LLMs, such as GEITje 7B Ultra, which are twice its size.

Key Capabilities

• Instruction-tuned for Dutch: Specialized in understanding and generating responses based on Dutch instructions.
• Efficient Performance: Achieves strong results for Dutch language tasks with a relatively small parameter count.
• Adaptation of Phi-2: Leverages the robust architecture of Phi-2, tailored for a specific linguistic context.

Training Details

The model was fine-tuned using a total of 201,579 samples from Dutch datasets, including BramVanroy/ultrachat_200k_dutch, BramVanroy/no_robots_dutch, and BramVanroy/belebele_dutch. Training was conducted using the alignment-handbook with DeepSpeed, utilizing 16 A100 80GB GPUs. The training process involved 3 epochs with a learning rate of 6e-05 and a total batch size of 672.

Good for

• Applications requiring Dutch language generation and instruction following.
• Developers seeking an efficient and performant LLM for Dutch-specific tasks.
• Research and development in Dutch natural language processing.