uzcaliskan/kth-tekop-sondaj-model

Model Overview

The uzcaliskan/kth-tekop-sondaj-model is an 8 billion parameter language model, fine-tuned from unsloth/qwen3-8b-unsloth-bnb-4bit by uzcaliskan. The fine-tuning process utilized Unsloth and Huggingface's TRL library, enabling faster training.

Key Capabilities and Performance

This model was evaluated on a specialized 100-question MMLU test set (uzcaliskan/kth-tekop-sondaj-mmlu-testi) tailored to KTH Tekop Directorate's drilling report format. In comparison with 6 other models, it achieved a success rate of 72-74% (with think=False and think=True respectively), placing it in the 4th-5th position among the tested models. The model demonstrates that its general language and reasoning abilities were preserved during fine-tuning, performing well in categories it had not explicitly seen during training, such as dst_log_operasyonu and sapma_acisi.

Noteworthy Findings

• General ability preserved: The model maintained its general language and reasoning capabilities, avoiding catastrophic forgetting during fine-tuning.
• Competitive in domain categories: It performs comparably to general-purpose base models (like gemma4, qwen3, qwen3.5) in categories directly related to its training data, such as centralizer, durum_tespiti, kuyudaki_casing, and kacak_durumu.
• Identified Weakness: A significant and isolated weakness was observed in the siradaki_casing category, where the model performed at 10-20% accuracy, below random chance. This suggests a potential issue in the training data or rule learning for this specific task.
• Limited thinking effect: The difference in performance between think=True and think=False was only 2 percentage points, indicating stable performance regardless of the thinking mode.

Limitations

• Small sample size: The evaluation used only 100 questions, leading to wider 95% confidence intervals and making small percentage differences less statistically significant.
• Narrow domain benchmark: The test is specific to drilling reports and does not generalize to overall model capabilities.
• Thinking mode disabled: The primary evaluation was conducted with think=False, which might underestimate the model's true performance if it was trained to utilize a thinking block.