ai4bharat/hercule-hi

TEXT GENERATIONPricing:Input $0.2 / Cached $0.028 / Output $0.32Concurrent Unit Cost:1Model Size:8BQuant:FP8Context Size:32kTool Calling:SupportedPublished:Sep 18, 2024License:mitArchitecture:Transformer Open Weights Featherless Exclusive Cold

Hercule-hi is an 8 billion parameter cross-lingual evaluation model developed by AI4Bharat, fine-tuned on Llama-3.1-8B-Instruct. It specializes in assessing multilingual Large Language Models (LLMs) by comparing their outputs to English reference responses. This model provides reference-based feedback and scores on a 1-5 scale, demonstrating strong alignment with human judgments, particularly in low-resource and zero-shot multilingual evaluation scenarios.

Loading preview...

Hercule-hi: Cross-Lingual LLM Evaluation

Hercule-hi is an 8 billion parameter evaluation model developed by AI4Bharat, specifically designed for assessing multilingual Large Language Models (LLMs). It is fine-tuned on the Llama-3.1-8B-Instruct architecture using the INTEL dataset.

Key Capabilities

  • Cross-Lingual Evaluation: Addresses the challenge of evaluating multilingual LLMs by using English reference responses to score multilingual outputs.
  • Human Judgment Alignment: Demonstrates better alignment with human judgments compared to zero-shot evaluations by proprietary models like GPT-4 on the RECON test set.
  • Low-Resource & Zero-Shot Support: Excels in low-resource scenarios and supports zero-shot evaluations on unseen languages.
  • Reference-Based Scoring: Provides detailed feedback and scores on a 1-5 scale, based on a reference answer and a scoring rubric.
  • Efficient Fine-tuning: Highlights the effectiveness of lightweight fine-tuning methods (like LoRA) for efficient multilingual evaluation.

Use Cases

Hercule-hi is ideal for developers and researchers who need to objectively evaluate the quality of multilingual LLM responses, especially when comparing them against a known good English reference. It's particularly useful for:

  • Automated quality assessment of LLM outputs in various languages.
  • Benchmarking multilingual LLMs against human-like judgments.
  • Evaluating model performance in languages with limited training data.

For more technical details, refer to the research paper and the GitHub repository, which also provides wrapper functions for easier integration.

Finetunes
3 models