staticlabs/dlm-code0.6b-exp
dlm-code0.6b-exp is an experimental 0.5 billion parameter masked-diffusion language model for code, developed by staticlabs as part of the thunder-fast project. Unlike traditional autoregressive models, it uses a discrete masked-diffusion (MDM) approach based on a Qwen2 backbone, generating code through a progressive un-masking loop with bidirectional attention. This model excels at parallel code generation within a window-based context and supports infilling, making it distinct from next-token prediction models.
Loading preview...
dlm-code0.6b-exp: Experimental Masked-Diffusion Code Model
dlm-code0.6b-exp is an experimental 0.5 billion parameter code model from the thunder-fast project, developed by staticlabs. It represents a significant departure from standard autoregressive language models by employing a discrete masked-diffusion (MDM) approach. Built on a Qwen2 backbone, this model generates code by progressively un-masking tokens over 24 diffusion steps, utilizing bidirectional attention to fill an entire output window in parallel.
Key Capabilities & Features
- Masked-Diffusion Generation: Utilizes a unique iterative un-masking loop for code generation, rather than left-to-right token prediction.
- Bidirectional Attention: Enables parallel filling of output windows, offering a different generation paradigm.
- Qwen2 Backbone: Based on a Qwen2 architecture with 24 layers, 896 hidden size, and GQA (14 attention / 2 KV heads).
- Context & Infilling: Operates with a window-based context of 256 tokens and supports infilling tasks.
- Special Mask Token: Incorporates a dedicated
<M>mask token (ID 151665) within its 151,936-entry vocabulary.
When to Use This Model
This model is specifically designed for use with a diffusion decoder, such as the thunder-fast runtime. It is not intended to be loaded as a standard causal Qwen2ForCausalLM for autoregressive generation. Developers interested in exploring novel, non-autoregressive code generation techniques, particularly those involving parallel un-masking and infilling, will find this experimental checkpoint valuable for research and development within the thunder-fast ecosystem.