ai4bharat/hercule-te

TEXT GENERATIONPricing:Input $0.2 / Cached $0.028 / Output $0.32Concurrent Unit Cost:1Model Size:8BQuant:FP8Context Size:32kTool Calling:SupportedPublished:Sep 18, 2024License:mitArchitecture:Transformer Open Weights Featherless Exclusive Cold

Hercule is an 8 billion parameter cross-lingual evaluation model developed by AI4Bharat, fine-tuned on Llama-3.1-8B-Instruct with a 32768 token context length. It is designed to assess multilingual Large Language Models (LLMs) by scoring multilingual outputs against English reference responses. The model excels in low-resource scenarios and supports zero-shot evaluations for unseen languages, providing feedback and scores on a 1-5 scale.

Loading preview...

Hercule: Cross-Lingual LLM Evaluation Model

Hercule, developed by AI4Bharat, is an 8 billion parameter evaluation model fine-tuned on Llama-3.1-8B-Instruct. It is specifically designed to address the challenges of evaluating multilingual Large Language Models (LLMs) by using English reference responses to score multilingual outputs. The model employs a reference-based evaluation approach, providing both qualitative feedback and a quantitative score on a 1-5 scale.

Key Capabilities

  • Cross-Lingual Evaluation: Assesses multilingual LLM responses by comparing them against English reference answers.
  • Low-Resource Language Support: Demonstrates strong performance in scenarios with limited linguistic data.
  • Zero-Shot Evaluation: Capable of evaluating unseen languages without prior training on those specific languages.
  • Human Alignment: Fine-tuned on the INTEL dataset, Hercule shows better alignment with human judgments on the RECON test set compared to zero-shot evaluations by proprietary models like GPT-4.
  • Efficient Fine-tuning: Highlights the effectiveness of lightweight fine-tuning methods (like LoRA) for efficient multilingual evaluation.

Use Cases

  • Automated LLM Assessment: Ideal for developers and researchers needing to automatically evaluate the quality of multilingual LLM outputs.
  • Benchmarking: Useful for benchmarking and comparing different multilingual LLMs, especially in low-resource language contexts.
  • Quality Assurance: Can be integrated into development pipelines to ensure the quality and accuracy of LLM responses across various languages.

For more details, refer to the research paper and the GitHub repository.

Finetunes
2 models