dharun2049/Brahmaputra-1
Brahmaputra-1 is a 0.5 billion parameter language model developed by dharun2049, distilled from Qwen/Qwen3.8-27B. It utilizes a compute-constrained contrastive knowledge-distillation (CKD) stage with a focus on multiple-choice question answering and incorporates Codeforces anti-forgetting replay for C++ coding preservation. This model is optimized for efficient performance in reasoning tasks while maintaining coding capabilities, making it suitable for applications requiring compact yet capable language understanding.
Loading preview...
Brahmaputra-1: A Compact, Distilled Language Model
Brahmaputra-1 is a 0.5 billion parameter model developed by dharun2049, built upon dharun2049/kaveri-stgrpo-0.5b. Its core innovation lies in a compute-constrained contrastive knowledge-distillation (CKD) process, leveraging Qwen/Qwen3.8-27B as a dense teacher model.
Key Capabilities & Distillation Process
- Knowledge Distillation: The model learns from the teacher's four-way answer probability geometry for multiple-choice questions, optimizing a loss function that includes supervised multiple-choice cross-entropy (
L_MC), temperature-scaled KL divergence (L_KD), a margin loss (L_margin), and an explanation loss (L_explanation). - Coding Anti-forgetting: To preserve coding abilities, the CKD stage interleaves approximately 15% C++ replay micro-batches from
open-r1/codeforces. This mechanism ensures that the model retains its C++ coding proficiency, as evidenced by code-preservation probes before and after CKD. - Confidence-Gated KD: Teacher knowledge distillation is confidence-gated; examples where the teacher's top choice disagrees with the gold answer receive zero KD weight, focusing learning on reliable teacher signals.
Training & Data
The model was trained using a LoRA adapter (rank 16, alpha 32) targeting various projection layers. Training data included sources like medmcqa, arc_challenge, openbookqa, arc_easy, and commonsenseqa, with a total of 7969 teacher-scored examples and 59 Qwen3.8-27B-generated explanations. Notably, MMLU dev/test sets were explicitly excluded from training.
Use Cases
Brahmaputra-1 is particularly well-suited for applications requiring a compact model with strong multiple-choice reasoning capabilities and preserved coding knowledge, making it efficient for deployment in resource-constrained environments.