allenai/digital-socrates-13b

What is Digital Socrates 13B?

Digital Socrates 13B (DS-13B) is a 13 billion parameter model developed by AllenAI, fine-tuned from the Llama-2-13b-Chat architecture. Its core purpose is to function as an open-source, automatic explanation-critiquing model. It analyzes explanations generated by other language models, providing nuanced, interpretable evaluations of their reasoning chains.

Key Capabilities

• Automatic Explanation Critiquing: Given a question, gold answer, and a student model's explanation and predicted answer, DS-13B generates a detailed critique.
• Localized Feedback: Identifies the most significant flaw in an explanation, if any, and pinpoints its location.
• Improvement Suggestions: Offers general and specific advice for revising flawed explanations.
• Numeric Quality Rating: Assigns an "Explanation score" to quantify the quality of the provided explanation.
• Cost-Effective Evaluation: Enables automatic evaluation of explanations without the need for expensive API calls to larger models or extensive human annotation.
• Performance: Despite being significantly smaller than models like GPT-4, Digital Socrates models generate critiques that are quantitatively and qualitatively comparable in terms of human ratings and error category matches.

Use Cases

• Evaluating LLM Reasoning: Ideal for researchers and developers looking to understand the nature and quality of explanations produced by various LLMs.
• Debugging Student Models: Provides insights into the reasoning processes of student models, helping to identify and address weaknesses.
• Automated Assessment: Can be integrated into systems requiring automatic, interpretable evaluation of explanatory text.

For more details, refer to the Digital Socrates paper and dataset.