SanjiWatsuki/openchat-3.5-1210-starling-slerp
SanjiWatsuki/openchat-3.5-1210-starling-slerp is a 7 billion parameter language model created by SanjiWatsuki, leveraging a Slerp merge of openchat/openchat-3.5-1210 and berkeley-nest/Starling-LM-7B-alpha. This model combines the strengths of OpenChat-3.5 variants, including those trained with Feedback-Collection and a de-contaminated Capybara dataset, and Starling's novel training methods. It is designed to retain the benefits of both foundational models, offering enhanced conversational and reasoning capabilities within a 4096 token context window.
Loading preview...
Model Overview
This model, developed by SanjiWatsuki, is a 7 billion parameter language model created using the Slerp merge method. It combines two prominent OpenChat-based models:
- openchat/openchat-3.5-1210: An OpenChat-3.5 variant enhanced with the Feedback-Collection dataset and a de-contaminated Capybara dataset.
- berkeley-nest/Starling-LM-7B-alpha: Another OpenChat-3.5 variant, distinguished by its training with a novel method on the Nectar dataset.
Key Characteristics
The Slerp merge aims to integrate the distinct advantages of both foundational models. The base model for this merge is openchat/openchat-3.5-1210. The merging process specifically applies different t values for self-attention and MLP layers, indicating a nuanced combination strategy rather than a simple average.
Intended Purpose
The creator's intention behind this merge is to produce a model that retains the benefits and improved performance characteristics of both openchat-3.5-1210 and Starling-LM-7B-alpha. This suggests a focus on robust conversational abilities and potentially improved reasoning, building on the advanced training methodologies of its components.