Model Overview
This model, kth8/Llama-3.2-3B-Instruct-SuperGPQA-Classifier, is a specialized fine-tune of the unsloth/Llama-3.2-3B-Instruct base model, featuring 3.2 billion parameters. Its primary function is to act as a classifier, categorizing problems into specific disciplines, fields, and subfields based on the SuperGPQA dataset.
Key Capabilities
- Structured Classification: Designed to output classifications in a precise JSON format, specifying
discipline, field, and subfield. - Academic/Technical Categorization: Optimized for problems spanning a wide range of academic and technical domains, including Philosophy, Science, Engineering, Medicine, and more.
- Instruction-Following: Benefits from its instruction-tuned base, making it responsive to classification prompts.
Training Details
The model was fine-tuned using PEFT (LoRA) with a rank of 32 and an alpha of 64, targeting q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj modules. Training involved 2 epochs with a batch size of 32, a learning rate of 0.0004, and an AdamW optimizer. It achieved an average training loss of 0.068 and a final validation loss of 0.050.
Usage Recommendation
For optimal classification results, it is recommended to set the inference temperature to 0.0.