iaa01/CIA-4B
The iaa01/CIA-4B is a 4 billion parameter language model post-trained with Reinforcement Learning using the ∆Belief reward signal. Developed by iaa01, this model is optimized for information-seeking tasks by reducing its own belief uncertainty over time, providing dense feedback and turn-level credit assignment. It learns general information-seeking strategies, demonstrated in environments like 'Twenty Questions', making it suitable for complex, open-ended queries.
Loading preview...
Model Overview
iaa01/CIA-4B is a 4 billion parameter language model that has undergone reinforcement learning (RL) post-training. Its key differentiator is the use of the ∆Belief reward signal, a novel approach that rewards the model for reducing its internal belief uncertainty over time, rather than relying solely on sparse success signals.
Key Capabilities
- Dense Feedback: The ∆Belief reward signal provides continuous, dense feedback, which is crucial for training in complex, long-horizon tasks.
- Turn-Level Credit Assignment: This method enables precise credit assignment at each turn, improving learning efficiency in sequential decision-making.
- Information-Seeking Optimization: The model is specifically optimized for open-ended information-seeking tasks, learning to ask questions and gather data effectively.
- Generalization: Trained in environments like 'Twenty Questions', it develops general information-seeking strategies that can generalize beyond its training environment.
Good For
- Complex Question Answering: Ideal for scenarios requiring the model to actively seek information to resolve uncertainty.
- Interactive Agents: Suitable for developing agents that need to learn and adapt their information-gathering strategies in dynamic environments.
- Research in RL and Information Theory: Provides a strong baseline for exploring novel reward signals and uncertainty reduction in language models.
For more technical details, refer to the ∆Belief-RL project page and the code repository.