thu-ml/STAIR-Llama-3.1-8B-DPO-3
STAIR-Llama-3.1-8B-DPO-3 is an 8 billion parameter language model developed by thu-ml, fine-tuned from Meta's Llama-3.1-8B-Instruct. This model utilizes the STAIR self-improvement framework, undergoing three iterations of DPO training using self-generated prompt-response pairs. It is designed to produce responses with explicit reasoning steps and a final answer, making it suitable for tasks requiring structured thought processes and safety-aligned outputs.
Loading preview...
Model Overview
STAIR-Llama-3.1-8B-DPO-3 is an 8 billion parameter language model from thu-ml, built upon Meta's Llama-3.1-8B-Instruct. It has been fine-tuned using the STAIR (Self-Training with Auxiliary Information Retrieval) framework, specifically undergoing three iterations of self-improvement through DPO (Direct Preference Optimization) training. This process involves the model generating its own prompt-response pairs from various sources for further refinement.
Key Capabilities
- Structured Reasoning: The model is designed to output responses that include explicit reasoning steps, followed by a distinct final answer. This structure is indicated by special tokens like
<|Reasoning_step|>and<|Output|>. - Safety Alignment: By separating reasoning from the final answer, the model facilitates evaluation of correctness and safety, focusing on the final answer for assessment.
- Self-Improvement: Leverages the STAIR framework for iterative self-training, enhancing its performance through generated data.
Use Cases
This model is particularly well-suited for applications where:
- Transparency in Reasoning is crucial, allowing users to follow the model's thought process.
- Safety and Ethical Considerations are paramount, as the structured output aids in extracting and evaluating final, potentially sensitive, answers.
- Complex Problem Solving benefits from a step-by-step approach to derive solutions.