rootcastleengineering/sofia-chat-distilled-v0.1

Sofia Chat Distilled v0.1 Overview

Sofia Chat Distilled v0.1 is an experimental English technical assistant developed by Rootcastle Engineering & Innovation. This model is a distilled version of the Qwen2.5-0.5B-Instruct architecture, specifically designed for technical question answering. It was created by pruning and distilling the teacher model, retaining its embeddings and tokenizer, and using a LoRA adapter trained with a product-rotation kernel alignment objective.

Key Characteristics & Performance

• Distilled Architecture: An 18-layer student model initialized from selected teacher blocks of Qwen2.5-0.5B-Instruct, resulting in 18.11% fewer parameters (404.5M vs 494M).
• Specialized Training: Trained on 192 original English prompts covering engineering, signal processing, Python, ML, and verifiable-kernel concepts, using assistant-token hard cross-entropy, temperature-scaled KL divergence, and a kernel alignment penalty.
• Improved Perplexity: Achieves a validation reference perplexity of 67.96 and test reference perplexity of 84.14, significantly outperforming the pruned baseline (6,635.52 and 10,327.21 respectively).
• Novel Objective: Incorporates a product-rotation kernel alignment objective, comparing fixed four-dimensional sketches of response hidden states against teacher landmarks.

Limitations and Use Cases

This model is a reproducible distillation experiment and not intended as a reliable general-purpose technical adviser. It was trained on a narrow pilot dataset, and observed quality limitations include factual errors and imprecise definitions in held-out generations. It is best suited for researchers and developers interested in exploring model distillation techniques, particularly those involving kernel alignment, and for focused experiments in technical Q&A within its specific training domain. It is not recommended for applications requiring broad general knowledge or high factual accuracy without further fine-tuning and validation.