ethicalabs/Kurtis-E1.1-Qwen2.5-3B-Instruct
Kurtis-E1.1-Qwen2.5-3B-Instruct is a 3.09 billion parameter instruction-tuned causal language model developed by ethicalabs, fine-tuned using the flower framework. This experimental model is based on the Qwen2.5 architecture and is primarily intended for academic evaluation and research. It demonstrates general language understanding capabilities, achieving an MMLU score of 0.6522 (0-shot) and 0.6629 (5-shot), making it suitable for exploring instruction-following tasks in a research context.
Loading preview...
Model Overview
ethicalabs/Kurtis-E1.1-Qwen2.5-3B-Instruct is an experimental instruction-tuned causal language model, fine-tuned by ethicalabs using the flower framework. This model is based on the Qwen2.5 architecture and is specifically designed for academic evaluation and research purposes.
Key Capabilities & Performance
Evaluated using the LM Evaluation Harness, Kurtis-E1.1-Qwen2.5-3B-Instruct demonstrates general language understanding and reasoning abilities:
- HellaSwag: Achieved an
acc_normof 0.7412 (0-shot). - ARC-Easy: Scored an
acc_normof 0.6789 (0-shot). - ARC-Challenge: Reached an
acc_normof 0.448 (0-shot). - MMLU: Demonstrated an overall
accof 0.6522 (0-shot) and improved to 0.6629 (5-shot), with notable performance in social sciences (0.7618 0-shot) and humanities (0.5734 0-shot).
Important Considerations
This model is strictly experimental and comes with critical constraints:
- No Production Deployment: It is not intended for commercial, enterprise, or mission-critical environments.
- No Liability: Provided "as-is" without warranties; developers assume zero liability for any consequences of unauthorized deployment.
Good for
- Academic Research: Ideal for researchers exploring instruction-tuned models and their performance characteristics.
- Experimental Evaluation: Suitable for evaluating the impact of fine-tuning methods on Qwen2.5-3B-Instruct.
- Benchmarking: Useful for comparing performance against other models in a controlled research setting.