xw1234gan/seccodeplt-qwen2.5-coder-3b-grpo-no-kl-real-reward-v2
The xw1234gan/seccodeplt-qwen2.5-coder-3b-grpo-no-kl-real-reward-v2 model is a 3.1 billion parameter Qwen2.5-Coder-3B-Instruct variant, fine-tuned using GRPO without KL regularization for secure code generation. It focuses on improving compliance and safety in generated code, specifically for the SecCodePLT+ experiment. This model is optimized to correct causal-label alignment and utilizes a ReaL safety-unit-test reward with DAPO-style token loss, making it suitable for applications requiring robust code safety and compliance.
Loading preview...
Model Overview
The seccodeplt-qwen2.5-coder-3b-grpo-no-kl-real-reward-v2 is a specialized 3.1 billion parameter model derived from Qwen/Qwen2.5-Coder-3B-Instruct. Developed by xw1234gan, this version was fine-tuned using the GRPO (Generative Reinforcement Learning with Policy Optimization) method, specifically without KL regularization, for the SecCodePLT+ compliance experiment.
Key Features and Training
- Reinforcement Learning: Utilizes GRPO for fine-tuning, focusing on improving code generation quality based on a reward signal.
- Causal-Label Alignment: Incorporates corrections for causal-label alignment during training.
- Reward Mechanism: Employs the official ReaL safety-unit-test reward, combined with DAPO-style token loss and dynamic sampling, to guide the learning process towards safer and more compliant code.
- Training Data: Trained on a 655-example training split with a specific seed (42).
Evaluation Highlights
Evaluated on 164 official test examples using greedy decoding, the model achieved notable results in specific areas:
- Output Format Pass: 96.95%
- Syntax Pass: 96.95%
- Safety Pass: 56.71%
- Joint Pass: 15.24%
Limitations
It is important to note that this is a single-seed research checkpoint. Its evaluation was conducted with a resource-bounded Python verifier specific to the benchmark, and therefore, it does not provide a general guarantee of secure code generation for all contexts.