ai4bharat/llama-prometheus

TEXT GENERATIONPricing:Input $0.2 / Cached $0.028 / Output $0.32Concurrent Unit Cost:1Model Size:8BQuant:FP8Context Size:32kTool Calling:SupportedPublished:Sep 28, 2024License:mitArchitecture:Transformer Open Weights Featherless Exclusive Cold

Llama-Prometheus is an 8 billion parameter English evaluation model developed by AI4Bharat, fine-tuned on the Feedback-Collection dataset. Built upon the Llama-3.1-8B-Instruct base, it is designed to assess the quality of responses from multilingual Large Language Models (LLMs). This model specializes in reference-guided direct assessment, providing detailed feedback and a score based on a provided instruction, response, reference answer, and a 5-point scoring rubric.

Loading preview...

Llama-Prometheus: An LLM Evaluator

Llama-Prometheus, developed by AI4Bharat, is an 8 billion parameter English evaluation model designed to assess the quality of responses generated by other Large Language Models (LLMs). It is part of the broader CIA Suite, which focuses on cross-lingual auto-evaluation for multilingual LLMs.

Key Capabilities and Features

  • Evaluation Model: Functions specifically as an evaluator, not a generative model for general tasks.
  • Base Model: Fine-tuned from Meta's Llama-3.1-8B-Instruct, leveraging its strong foundational capabilities.
  • Training: Utilizes the Feedback-Collection dataset, mirroring the setup of Prometheus 2 for consistent evaluation methodology.
  • Reference-Guided Assessment: Employs a structured prompt format requiring an instruction, the response to evaluate, a reference answer (score 5), and a detailed 5-point scoring rubric.
  • Output Format: Generates detailed feedback followed by an integer score (1-5), separated by [RESULT].
  • Integration: Wrapper functions and classes are available on the GitHub repository to simplify its use.

Use Cases

Llama-Prometheus is ideal for developers and researchers who need to:

  • Automate LLM Evaluation: Systematically assess the quality of LLM outputs against specific criteria and reference answers.
  • Benchmark LLMs: Compare the performance of different LLMs or different versions of the same LLM based on consistent evaluation metrics.
  • Integrate into CI/CD: Incorporate automated quality checks for LLM-powered applications during development workflows.

Finetunes
1 models