srivarenya/MoM-python-slm
srivarenya/MoM-python-slm is a 1.5 billion parameter Python code-generation node, part of a Mixture-of-Models (MoM) mesh, built upon Qwen2.5-Coder-1.5B-Instruct with a 32768 token context length. This specialized model focuses on single-turn Python code generation, including reasoning, and is optimized for library/API capability. It achieves 70.7 pass@1 on HumanEval and 69.6 pass@1 on MBPP, demonstrating strong performance in generating correct code from specifications.
Loading preview...
MoM-Python-SLM: Specialized Python Code Generation
This model, developed by srivarenya, is a 1.5 billion parameter Small Language Model (SLM) specifically designed for Python code generation. It functions as a specialized node within a larger Mixture-of-Models (MoM) mesh, aiming to surpass generalist models in coding tasks through deep specialization rather than sheer parameter count. Built on the Qwen2.5-Coder-1.5B-Instruct base, it shares a common tokenizer for logit-space fusion across the MoM mesh.
Key Capabilities & Training
- Single-Turn Code Generation: Given a Python task, it provides reasoning followed by code, optionally using an upstream context packet.
- Specialized Training: Fine-tuned using DoRA (r=64) on 476K instances, including CPython documentation, Flask/Requests source, issues, CVEs, and execution-verified synthetic problems. The dataset is decontaminated against HumanEval/MBPP.
- Strong Performance: Achieves 70.7 pass@1 on HumanEval and 69.6 pass@1 on MBPP, outperforming its base model. It shows significant gains in library/API capability, such as writing correct code from specifications and recalling API signatures.
Usage & Future Directions
This model is intended for direct integration into Python environments using the Hugging Face transformers library. Future development includes GRPO/RLVR against execution-grounded rewards to further enhance performance beyond instruct-tuning.