Technoculture/BioMistral-Hermes-Slerp

TEXT GENERATIONConcurrency Cost:1Model Size:7BQuant:FP8Ctx Length:4kPublished:Feb 21, 2024License:apache-2.0Architecture:Transformer Open Weights Cold

Technoculture/BioMistral-Hermes-Slerp is a 7 billion parameter language model created by Technoculture, merging BioMistral/BioMistral-7B-DARE and NousResearch/Nous-Hermes-2-Mistral-7B-DPO using the slerp method. This model is specifically designed to combine the biomedical knowledge of BioMistral with the general conversational and instruction-following capabilities of Nous-Hermes-2. With a 4096-token context length, it aims to provide a versatile foundation for applications requiring both specialized medical understanding and broad language proficiency.

Loading preview...

BioMistral-Hermes-Slerp: A Merged Language Model

BioMistral-Hermes-Slerp is a 7 billion parameter language model developed by Technoculture, created by merging two distinct models: BioMistral/BioMistral-7B-DARE and NousResearch/Nous-Hermes-2-Mistral-7B-DPO. This merge was performed using the slerp (spherical linear interpolation) method, aiming to combine their respective strengths.

Key Characteristics

  • Hybrid Knowledge Base: Integrates the biomedical domain expertise from BioMistral-7B-DARE with the general instruction-following and conversational abilities of Nous-Hermes-2-Mistral-7B-DPO.
  • Merge Method: Utilizes the slerp technique, with specific parameter weighting applied to different layers (e.g., self_attn and mlp) to fine-tune the blend of the source models.
  • Base Architecture: Built upon the Mistral 7B architecture, inheriting its efficiency and performance characteristics.

Potential Use Cases

  • Biomedical Q&A: Answering questions related to medical concepts, research, and clinical information.
  • Healthcare Applications: Developing tools for medical text summarization, patient interaction, or educational content generation.
  • General Conversational AI: Engaging in broad discussions while retaining the ability to pivot to specialized biomedical topics.
  • Research and Development: Serving as a foundation for further fine-tuning on specific biomedical or general-purpose tasks where a blend of knowledge is beneficial.

Finetunes
1 models