ai4bharat/hercule-fr

Hugging Face
TEXT GENERATIONPricing:Input $0.2 / Cached $0.028 / Output $0.32Concurrent Unit Cost:1Model Size:8BQuant:FP8Context Size:32kTool Calling:SupportedPublished:Sep 18, 2024License:mitArchitecture:Transformer Open Weights Featherless Exclusive Warm

ai4bharat/hercule-fr is an 8 billion parameter cross-lingual evaluation model, fine-tuned on Llama-3.1-8B-Instruct, designed to assess multilingual Large Language Models (LLMs). Developed by AI4Bharat, it uses English reference responses to score multilingual outputs, demonstrating strong alignment with human judgments, particularly in low-resource scenarios. This model supports zero-shot evaluations on unseen languages and provides feedback and scores on a 1-5 scale, making it ideal for efficient and accurate multilingual LLM assessment.

Loading preview...

Hercule-fr: Cross-Lingual LLM Evaluation Model

Hercule-fr is an 8 billion parameter evaluation model developed by AI4Bharat, specifically fine-tuned on Llama-3.1-8B-Instruct using the INTEL dataset. Its primary purpose is to assess multilingual Large Language Models (LLMs) by leveraging English reference responses to score multilingual outputs, addressing a key challenge in multilingual LLM evaluation.

Key Capabilities

  • Cross-Lingual Evaluation: Designed to evaluate LLMs across different languages, using English as a reference for scoring.
  • Human Alignment: Demonstrates better alignment with human judgments compared to zero-shot evaluations by proprietary models like GPT-4 on the RECON test set.
  • Low-Resource Performance: Excels particularly in scenarios involving low-resource languages.
  • Zero-Shot Evaluation: Capable of performing zero-shot evaluations on unseen languages.
  • Reference-Based Scoring: Provides detailed feedback and scores on a 1-5 scale, based on a provided scoring rubric and reference answer.
  • Efficient Fine-tuning: Highlights the effectiveness of lightweight fine-tuning methods (like LoRA) for efficient multilingual evaluation.

Use Cases

Hercule-fr is ideal for developers and researchers who need to:

  • Evaluate multilingual LLMs: Accurately assess the performance of LLMs that generate content in various languages.
  • Benchmark LLMs: Compare the quality of different LLM outputs using a standardized, human-aligned scoring mechanism.
  • Develop multilingual applications: Ensure the quality and relevance of LLM-generated content in diverse linguistic contexts.

For ease of use, wrapper functions and classes are available on the AI4Bharat GitHub repository. The model follows a specific prompt format, similar to Prometheus 2, requiring an instruction, response to evaluate, English reference answer, and a detailed scoring rubric.

Finetunes
2 models