Model Overview

This model, developed by SanjiWatsuki, is a 7 billion parameter language model created using the Slerp merge method. It combines two prominent OpenChat-based models:

• openchat/openchat-3.5-1210: An OpenChat-3.5 variant enhanced with the Feedback-Collection dataset and a de-contaminated Capybara dataset.
• berkeley-nest/Starling-LM-7B-alpha: Another OpenChat-3.5 variant, distinguished by its training with a novel method on the Nectar dataset.

Key Characteristics

The Slerp merge aims to integrate the distinct advantages of both foundational models. The base model for this merge is openchat/openchat-3.5-1210. The merging process specifically applies different t values for self-attention and MLP layers, indicating a nuanced combination strategy rather than a simple average.

Intended Purpose

The creator's intention behind this merge is to produce a model that retains the benefits and improved performance characteristics of both openchat-3.5-1210 and Starling-LM-7B-alpha. This suggests a focus on robust conversational abilities and potentially improved reasoning, building on the advanced training methodologies of its components.