NickyNicky/Mistral-7B-OpenOrca-oasst_top1_2023-08-25-v2
NickyNicky/Mistral-7B-OpenOrca-oasst_top1_2023-08-25-v2 is a 7 billion parameter Mistral-based causal language model developed by NickyNicky. This model is fine-tuned on the OpenAssistant/oasst_top1_2023-08-25 dataset and incorporates attention sink technology for improved generation efficiency. It is designed for general-purpose conversational AI and instruction following, supporting multilingual interactions across 20 languages.
Loading preview...
Model Overview
NickyNicky/Mistral-7B-OpenOrca-oasst_top1_2023-08-25-v2 is a 7-billion parameter language model built upon the Mistral architecture, further fine-tuned from the Open-Orca/Mistral-7B-OpenOrca base model. This iteration specifically leverages the OpenAssistant/oasst_top1_2023-08-25 dataset, which includes multilingual data across 20 languages, enhancing its conversational and instruction-following capabilities.
Key Features & Enhancements
- Base Model: Fine-tuned from
Open-Orca/Mistral-7B-OpenOrca. - Training Data: Utilizes the
OpenAssistant/oasst_top1_2023-08-25dataset for instruction tuning, covering a broad range of languages (e.g., English, Spanish, German, French, Russian). - Attention Sinks: Incorporates the
attention_sinkstechnique, which can improve generation efficiency and context handling, particularly for longer sequences. This is configured with anattention_sink_sizeof 4 andattention_sink_window_sizeof 1024. - Multilingual Support: Benefits from the diverse language coverage of the OASTT dataset, making it suitable for multilingual applications.
Usage Considerations
This model is designed for tasks requiring instruction following and conversational generation. The integration of attention sinks suggests potential benefits for maintaining coherence over extended dialogues or complex instructions. Example usage demonstrates its ability to generate code snippets and respond to queries in multiple languages.