Shuibai12138/Open-Dcoder-0.5B-mixture-mdm-step2000
Shuibai12138/Open-Dcoder-0.5B-mixture-mdm-step2000 is a 0.5 billion parameter corrective diffusion language model (CDLM-0.5B) based on the Qwen2 architecture, developed by Shuibai Zhang and others. This model is a continuation of fredzzp/open-dcoder-0.5B, trained for 2,000 additional steps using a corrective diffusion objective. It is specifically designed as a masked diffusion language model with bidirectional attention, intended for use with a specialized evaluation pipeline for diffusion models rather than standard causal language model inference.
Loading preview...
Model Overview
Shuibai12138/Open-Dcoder-0.5B-mixture-mdm-step2000, also known as CDLM-0.5B, is a 0.5 billion parameter corrective diffusion language model. It is a continuation of the fredzzp/open-dcoder-0.5B model, further trained for 2,000 steps using a specific Corrective Diffusion Language Model (CDLM) objective, as detailed in the NeurIPS 2026 paper "Corrective Diffusion Language Models" by Zhang et al.
Key Characteristics
- Architecture: Based on Qwen2, but functions as a masked diffusion language model with bidirectional attention.
- Training Objective: Utilizes a CDLM objective involving absorbing corruption and uniform replacement of 10% of visible tokens, with a cross-entropy term on replaced positions.
- Training Data: Continued training on a subset of the
nvidia/Nemotron-Pretraining-SFT-v1dataset, specificallyNemotron-SFT-Code. - Specialized Usage: Unlike standard causal language models, this model requires a dedicated evaluation pipeline from the CDLM code repository for correct inference, as loading with
AutoModelForCausalLMwill produce incorrect outputs due to its bidirectional attention and shifted logits.
Intended Use Cases
This model is primarily intended for research and evaluation within the context of corrective diffusion language models. Developers interested in exploring diffusion-based language generation or replicating the results from the "Corrective Diffusion Language Models" paper should use this model with the provided evaluation pipeline.