Geraldine/Gemini-Distill-Qwen2.5-0.5B-ead

Hugging Face
TEXT GENERATIONPricing:Input $0.04 / Cached $0.008 / Output $0.08Concurrent Unit Cost:1Model Size:0.5BQuant:BF16Context Size:32kTool Calling:SupportedPublished:Mar 5, 2025License:mitArchitecture:Transformer Open Weights Featherless Exclusive Warm

Geraldine/Gemini-Distill-Qwen2.5-0.5B-ead is a 0.5 billion parameter language model fine-tuned from Qwen2.5-0.5B-Instruct. It specializes in generating Encoded Archival Description (EAD/XML) outputs, utilizing a structured reasoning approach distilled from Gemini-2.0-Flash-Thinking-Exp. The model first provides detailed reasoning and then outputs the final EAD/XML response, enhancing interpretability and accuracy for archival metadata tasks. It supports a context length of 32768 tokens and is multilingual, covering languages like Chinese, English, French, Spanish, and more.

Loading preview...

Model Overview

Geraldine/Gemini-Distill-Qwen2.5-0.5B-ead is a specialized 0.5 billion parameter language model, fine-tuned from Qwen2.5-0.5B-Instruct. Its core capability lies in generating Encoded Archival Description (EAD/XML) outputs, a critical format for archival records metadata.

Key Capabilities & Features

  • EAD/XML Generation: Specifically trained to produce structured EAD/XML, essential for archivists and digital libraries.
  • Reasoning-First Approach: Employs a unique two-step process where it first generates detailed reasoning, then follows with the final EAD/XML output. This enhances transparency and accuracy.
  • Knowledge Distillation: The model's EAD/XML generation and reasoning capabilities were distilled from the more powerful Gemini-2.0-Flash-Thinking-Exp, allowing a smaller model to achieve specialized performance.
  • Multilingual Support: Trained on a dataset covering numerous languages including Chinese, English, French, Spanish, Portuguese, German, Italian, Russian, Japanese, Korean, Vietnamese, Thai, and Arabic.
  • Context Length: Supports a substantial context window of 32768 tokens, enabling processing of longer archival descriptions.

Training Details

The model was trained on the Geraldine/Ead-Instruct-4k-Distilled dataset, comprising 4,000 distilled samples. The training process focused on teaching the model to map user prompts to Gemini-generated reasoning traces and then to the final EAD/XML response. Hyperparameters included a batch size of 4, a max sequence length of 4096 tokens, and 5 epochs, utilizing bf16 precision and gradient checkpointing for efficiency.

Limitations

  • The current dataset size of 4,000 samples may limit generalization to highly diverse or complex EAD/XML scenarios.
  • Users should disable Sliding Window Attention (model.config.sliding_window = None) to prevent potential inference slowdowns.

Finetunes
3 models