Havoc999/tiny-openhermes
Havoc999/tiny-openhermes is a 1.1 billion parameter causal language model, fine-tuned by Havoc999 on the OpenHermes-2.5 dataset using LoRA. Based on TinyLlama/TinyLlama-1.1B-Chat-v1.0, this model offers a compact solution for general conversational AI with a 2048-token context length. It demonstrates moderate capabilities in physical commonsense reasoning, making it suitable for applications requiring efficient, smaller-scale language understanding.
Loading preview...
Tiny OpenHermes: A Compact Conversational Model
Havoc999/tiny-openhermes is a 1.1 billion parameter language model, fine-tuned from TinyLlama/TinyLlama-1.1B-Chat-v1.0 using LoRA (rank 32) on the teknium/OpenHermes-2.5 dataset. This model was trained for 1 epoch on Kaggle Dual T4 GPUs, focusing on conversational capabilities.
Key Capabilities & Performance
This model, despite its small size, exhibits baseline performance in common NLP tasks. It is primarily English-focused due to its training data. Benchmark results using the Language Model Evaluation Harness indicate:
- PIQA (Physical Commonsense): 72.03% normalized accuracy
- HellaSwag (Commonsense Reasoning): 59.20% normalized accuracy
- ARC-Challenge (Advanced Science): 29.69% normalized accuracy
- MMLU (Mathematics): 26.13% accuracy
Limitations
As a 1.1 billion parameter model, it has inherent limitations:
- English-primary: Predominantly English due to the OpenHermes-2.5 dataset.
- Hallucination: May hallucinate facts; verification of important claims is necessary.
- Complex Reasoning: Can struggle with complex, multi-step reasoning tasks.
- Alignment: Its safety alignment is based on TinyLlama's base alignment and is not further RLHF-aligned.