arrochi112/onebee-gf-distill-v1
arrochi112/onebee-gf-distill-v1 is a 5.1 billion parameter Gemma4-based multimodal language model, developed by arrochi112 as part of the small-mind-companion project. This model is post-trained via LoRA SFT, DPO, and on-policy distillation, specifically designed to function as a memory-augmented companion. It excels at conversational responses grounded in external memory retrieval, rather than relying on raw context length or parametric recall, and demonstrates strong calibration with a 71.25% UAR on the Personalized Memory Benchmark.
Loading preview...
Model Overview
arrochi112/onebee-gf-distill-v1 is the leading checkpoint from the small-mind-companion research project by arrochi112. This project investigates how small, vision-capable language models (around 2 billion effective parameters) can recover apparent capabilities through advanced post-training and external memory integration, rather than relying solely on parameter scale.
Key Capabilities
- Memory-Augmented Companion: Designed to retrieve facts from an external memory store and answer questions grounded in that memory, acting as a conversational companion.
- Advanced Post-Training: Utilizes a unique training chain: LoRA SFT (2232 examples) → LoRA DPO (2049 preference pairs) → on-policy distillation (2008 prompts, 125 steps).
- On-Policy Distillation: Features a distinct on-policy distillation stage applied after DPO, where the student model generates its own completions matched to a larger 8B Gemma teacher's distribution via generalized JSD.
- Strong Calibration: Achieves a 71.25% UAR (correctly abstains on unanswerable questions without over-hedging) and 18.59%
pra_lenientaccuracy on the Personalized Memory Benchmark (PMB). - Persona Consistency: Maintains or improves persona consistency post-distillation, as evidenced by LLM-judge pairwise comparisons and an independent stylometric self-consistency measure.
Good For
- Companion-Persona Conversational Models: Ideal for integration into memory/retrieval pipelines to create conversational agents that respond based on retrieved information.
- Post-Training Research: Serves as a valuable reference point for further research into distillation, quantization, and ablation studies on small multimodal models.
- Memory-Augmented LLM Development: Useful for developers exploring alternatives to increasing raw parameter scale or context length for enhanced factual grounding.