MD-Judge-v0.1: An LLM-Based Safety Classifier

MD-Judge-v0.1, developed by OpenSafetyLab, is a 7 billion parameter language model built upon the Mistral-7B architecture. Its primary function is to act as a safety guard, classifying the safety of question-answer (QA) pairs. This model is a core component of the 🥗SALAD-Bench framework, which aims to provide a comprehensive and hierarchical safety benchmark for large language models.

Key Capabilities & Features

• Safety Classification: Evaluates QA pairs against predefined safety policies, categorizing content as 'safe' or 'unsafe'.
• Detailed Unsafe Categories: Identifies specific violations across categories such as Representation & Toxicity, Misinformation Harms, Socioeconomic Harms, Information & Safety, Malicious Use, and Human Autonomy & Integrity.
• Superior Performance: Outperforms other methods like Keyword, LlamaGuard, GPT-3.5, and GPT-4 on various public safety test sets (ToxicChat, Beavertails, SafeRLHF, SALAD-Base-Test, SALAD-Enhance-Test) in F1 scores.
• Evaluation Tool: Designed to serve as a general evaluation tool for assessing the safety of different LLMs.

Use Cases

MD-Judge-v0.1 is ideal for developers and researchers focused on:

• Automated Content Moderation: Classifying and filtering potentially unsafe AI-generated content.
• LLM Safety Benchmarking: Evaluating the safety performance of other large language models within the SALAD-Bench framework.
• Developing Safer AI Systems: Integrating into AI pipelines to ensure outputs adhere to safety guidelines and prevent harmful responses.

For more technical details, including the research paper, code, and dataset, refer to the SALAD-Bench project page and GitHub repository.