yyqoni/Phi-3-mini-4k-segment-ppo-60k
The yyqoni/Phi-3-mini-4k-segment-ppo-60k is a 4 billion parameter language model, fine-tuned from microsoft/Phi-3-mini-4k-instruct, that utilizes a novel segment-level reward model for improved Reinforcement Learning with Human Feedback (RLHF). This model assigns rewards to semantically complete text segments based on entropy thresholds, enhancing the granularity and stability of RLHF training. It is designed to improve language model performance through more precise reward signaling during PPO training.
Loading preview...
Overview
The yyqoni/Phi-3-mini-4k-segment-ppo-60k model is a 4 billion parameter language model, fine-tuned from microsoft/Phi-3-mini-4k-instruct. Its core innovation lies in applying a segment-level reward model to Reinforcement Learning with Human Feedback (RLHF), as detailed in the paper "Segmenting Text and Learning Their Rewards for Improved RLHF in Language Model". This approach redefines RLHF granularity by assigning rewards to semantically complete text segments, identified using entropy thresholds, and introduces techniques to stabilize training under these dense, segment-level rewards.
Key Capabilities
- Segment-level RLHF: Utilizes a novel method to assign rewards to specific text segments, rather than entire responses, for more precise feedback.
- Improved Training Stability: Incorporates techniques to maintain stability during RLHF training despite the increased density of segment-level rewards.
- Enhanced Reward Modeling: Leverages entropy thresholds to define semantically complete text segments, enabling a more granular reward signal.
Training Details
The model was trained using a combination of datasets:
- Preference-700K Dataset: A diverse collection including HH-RLHF, Stanford Human Preferences Dataset (SHP), and HelpSteer.
- Ultrafeedback Dataset: Used specifically for sampling prompts during the PPO training routine.
Usage
This model can be directly integrated using the Hugging Face Transformers library for text generation tasks, benefiting from its refined RLHF training.