yyqoni/Phi-3-mini-4k-token-ppo-60k
yyqoni/Phi-3-mini-4k-token-ppo-60k is a 4 billion parameter language model based on the Phi-3-mini architecture, fine-tuned using a token-wise reward-based PPO method. This model incorporates techniques from the "Segmenting Text and Learning Their Rewards for Improved RLHF in Language Models" preprint. It is designed to leverage dense reward signals for enhanced Reinforcement Learning from Human Feedback (RLHF) in language generation tasks, offering a 4096-token context length.
Loading preview...
Overview
yyqoni/Phi-3-mini-4k-token-ppo-60k is a 4 billion parameter language model built upon the Phi-3-mini architecture. Its key differentiator lies in its fine-tuning approach, which utilizes a token-wise reward-based Proximal Policy Optimization (PPO) method. This technique is detailed in the preprint "Segmenting Text and Learning Their Rewards for Improved RLHF in Language Models" (https://arxiv.org/abs/2501.02790).
Key Capabilities
- Dense Reward RLHF: Employs a novel token-wise reward mechanism for Reinforcement Learning from Human Feedback, potentially leading to more granular and effective alignment.
- Phi-3-mini Foundation: Benefits from the compact yet capable architecture of Phi-3-mini, offering a balance of performance and efficiency.
- Context Length: Supports a context window of 4096 tokens, suitable for processing moderately long inputs.
Good For
- Researchers and developers interested in advanced RLHF techniques, particularly those involving dense reward signals.
- Applications requiring a compact model with improved alignment through sophisticated PPO fine-tuning.
- Experimentation with models that integrate segmenting text and learning rewards for enhanced language model performance. More details can be found in the associated repository: https://github.com/yinyueqin/DenseRewardRLHF-PPO.