KaraKaraWitch/BlenderCartel-llama33-70B-Pt2

Model Overview

KaraKaraWitch/BlenderCartel-llama33-70B-Pt2 is a 70 billion parameter merged language model, developed by KaraKaraWitch. It was constructed using the SCE (Sparse Component Ensemble) merge method, with deepcogito/cogito-v2-preview-llama-70B serving as its base model.

Key Capabilities & Merged Components

This model integrates a diverse set of capabilities by combining fourteen different Llama-3 and Llama-3.1 based models. The merge specifically targets a broad range of applications, including:

• Tool Calling: Incorporates watt-ai/watt-tool-70B for enhanced tool interaction capabilities.
• Multilingual Support: Includes models like rinna/llama-3-youko-70b (Japanese), yentinglin/Llama-3-Taiwan-70B-Instruct (Traditional Chinese), Bllossom/llama-3-Korean-Bllossom-70B (Korean), and FreedomIntelligence/AceGPT-v2-70B (Arabic), aiming for robust performance across multiple languages.
• Instruction Following: Integrates various instruction-tuned models such as kldzj/Llama-3.3-70B-Instruct-heretic and flammenai/Llama3.1-Flammades-70B.
• Diverse General Knowledge: Blends models like Delta-Vector/Shimamura-70B, Undi95/Sushi-v1.4, and Mawdistical/Anthrobomination-70B to enhance general understanding and response generation.

Merge Configuration

The merge process utilized a select_topk value of 0.2 and applied normalization, with the model weights stored in bfloat16 data type. This configuration aims to synthesize the strengths of the constituent models into a versatile and capable LLM.