Hugging Face Overview
uukuguy/speechless-zephyr-code-functionary-7b is a 7 billion parameter model built upon the Mistral-7B-v0.1 architecture. It represents an experimental approach called "Mixture-of-Multi-LoRAs" (moloras), where specialized LoRA modules are extracted and combined. This model integrates modules from three distinct sources:
- HuggingFaceH4/zephyr-7b-beta: Contributes to general conversational abilities.
- meetkai/functionary-small-v2.2: Enhances the model's capacity for executing functions and using plugins.
- uukuguy/speechless-code-mistral-7b-v1.0: Specifically designed to improve coding performance.
This dynamic combination is managed by a router that automatically assembles LoRA modules using a gradient-free approach, requiring minimal inference steps for new tasks. The entire process leverages extract-lora, merge-lora, and lora-hub tools from the multi-loras framework.
Performance
On the Open LLM Leaderboard, the model achieves an average score of 62.93, with notable scores including 61.52 on ARC, 83.88 on HellaSwag, 64.71 on MMLU, and 43.82 on GSM8K.
Good for
- Applications requiring a blend of general conversational skills, code generation, and function calling.
- Developers interested in exploring multi-LoRA architectures and dynamic model composition.
- Tasks benefiting from enhanced coding capabilities and plugin integration.