raicrits/Hermes7b_ITA: Italian Instruction-Following LLM

This model is a 7 billion parameter Large Language Model (LLM) developed by Stefano Scotta, specifically fine-tuned to understand and respond to instructions in Italian. It is based on the LLaMa2 architecture, with its foundation being the NousResearch/Nous-Hermes-llama-2-7b model, which itself is an instruction-tuned version of meta-llama/Llama-2-7b.

Key Capabilities

• Italian Instruction Following: The primary strength of Hermes7b_ITA is its ability to process and generate responses to instructions provided in Italian.
• Fine-tuning Potential: While capable of responding to simple instructions out-of-the-box, the model is also suitable for further fine-tuning to adapt it to more specific Italian-language tasks.
• LoRA Fine-tuning: The model was fine-tuned using the efficient LoRA (Low-Rank Adaptation) approach, merging the LoRA adapters with the base model weights.

Training Details

The fine-tuning process involved using 120,000 random instruction/answer pairs from the raicrits/Orca_ITA_200k dataset. Training was conducted over 3 epochs with a learning rate of 2e-4, utilizing mixed-precision training (float16) and a LoRA configuration targeting q_proj and v_proj modules.

Limitations

As with other LLMs, Hermes7b_ITA may generate content that is not factual, or produce biased, offensive, or inappropriate responses.