sangyon/llama31_tulu3_8b_dpo_grpo_nonthink_intentcheck_tulu3_8b_dpo
The sangyon/llama31_tulu3_8b_dpo_grpo_nonthink_intentcheck_tulu3_8b_dpo is an 8 billion parameter language model, based on the Llama-3.1-Tulu-3-8B-DPO architecture, fine-tuned using GRPO (Global Reward Policy Optimization). This model incorporates an "IntentCheck" judge and a programmatic constraint reward system, designed to enhance response quality and adherence to specific intentions. It is optimized for generating responses that pass an intent verification process, making it suitable for applications requiring controlled and intent-aligned text generation within its 32768 token context length.
Loading preview...
Model Overview
The sangyon/llama31_tulu3_8b_dpo_grpo_nonthink_intentcheck_tulu3_8b_dpo is an 8 billion parameter language model derived from the allenai/Llama-3.1-Tulu-3-8B-DPO base. It has been fine-tuned using a Global Reward Policy Optimization (GRPO) approach over four epochs, with a batch size of 1024 and eight rollouts per prompt.
Key Features and Training
- Base Model:
allenai/Llama-3.1-Tulu-3-8B-DPO. - Fine-tuning Method: GRPO, incorporating an "IntentCheck" judge.
- Reward Mechanism: Utilizes a programmatic constraint reward, with an additional
0.1bonus for responses that successfully pass the IntentCheck. - Context Length: Supports a maximum response length of 2048 tokens during training.
- Actor KL Coefficient: Set to
0.001during the GRPO training process. - Export Format: Merged into BF16 safetensors across four FSDP model shards.
Intended Use Cases
This model is particularly suited for applications where generated text needs to adhere to specific intentions or pass a verification check. Its training methodology, which includes an IntentCheck judge and a bonus for passing responses, aims to produce more controlled and aligned outputs. Developers can leverage this model for tasks requiring intent-aware response generation, where the quality is evaluated against predefined criteria.