Overview

Phigments12 is a 3 billion parameter language model developed by liminerity. It is constructed through a merge of two existing models, liminerity/merge6 and liminerity/merge3, utilizing the mergekit tool. This merging approach aims to combine the distinct characteristics and capabilities of its base models into a single, optimized unit.

Key Characteristics

• Merge Architecture: Built by merging liminerity/merge6 and liminerity/merge3.
• Merge Method: Employs the slerp (spherical linear interpolation) merge method.
• Layer-Specific Weighting: The merge configuration applies specific t values for self_attn and mlp layers, indicating a fine-tuned approach to combining the model components. This suggests an effort to balance or enhance particular aspects of the merged model's performance.
• Parameter Count: Operates with 3 billion parameters, making it a relatively compact model suitable for applications where resource efficiency is important.
• Context Length: Supports a context window of 2048 tokens.

Potential Use Cases

Given its merge-based origin and compact size, Phigments12 is likely suitable for tasks requiring efficient language processing where the combined strengths of its base models are beneficial. Its specific merge configuration might make it perform well in areas where self_attn and mlp layer characteristics are critical.