jeffmeloy/Qwen2.5-7B-nerd-uncensored-v1.3

Model Overview

This model, jeffmeloy/Qwen2.5-7B-nerd-uncensored-v1.3, is a 7.6 billion parameter Qwen2.5-based language model. Its unique characteristic lies in its construction method: it is a composite model where individual layers are programmatically selected from a pool of other Qwen2.5-7B models.

Key Construction Method: Normalized Effective Rank (NER)

The model's layers are chosen based on their Normalized Effective Rank (NER). This metric quantifies the dimensional utilization efficiency of each layer's weight matrix. The process involves:

• Computing singular values for each layer's weight matrix.
• Converting these singular values into a probability distribution.
• Calculating the Shannon entropy of this distribution.
• Normalizing the entropy to produce an NER score between 0 and 1.

Composite Model Creation

The ner_merge.py script (available in the v1.0 repository) automates the creation of this composite model. It downloads selected Qwen2.5-7B models, calculates the NER for each layer, and then incrementally builds the final model by selecting the layer with the highest NER score from the available pool for each position. This approach aims to integrate the most "information-rich" components from various fine-tuned Qwen2.5 variants.

Potential Use Cases

This model is suitable for users looking for a Qwen2.5-7B variant that potentially combines the strengths of multiple fine-tuned models without explicit re-training. Its construction method suggests an optimization for leveraging diverse learned representations, which could be beneficial for general-purpose tasks where robustness across various domains is desired.