trinhkhng/della_Merged_Qwen2-0.5B_0.1
trinhkhng/della_Merged_Qwen2-0.5B_0.1 is a 0.5 billion parameter language model merged using the DELLA method, based on Qwen2-0.5B. This model incorporates a debiased Qwen2-0.5B variant, aiming to integrate specific characteristics from the debiased model. It is designed for applications requiring a compact model with a 32768 token context length, benefiting from a specialized merging technique.
Loading preview...
Overview
trinhkhng/della_Merged_Qwen2-0.5B_0.1 is a 0.5 billion parameter language model created by trinhkhng through a merge of pre-trained models using the MergeKit tool. The base model for this merge is Qwen2-0.5B, and it specifically integrates a debiased version of Qwen2-0.5B.
Merge Details
This model utilizes the DELLA merge method, as described in the paper "DELLA: Debiasing Language Models with Low-Rank Adaptation". The merge configuration involved a density of 0.5 and an epsilon of 0.1 for the debiased model, with a weight of 1.0. The process also included int8 masking, normalization, and rescaling parameters.
Key Characteristics
- Parameter Count: 0.5 billion parameters, making it a compact model suitable for resource-constrained environments.
- Context Length: Supports a substantial context window of 32768 tokens.
- Merge Method: Employs the DELLA method, which is designed for integrating specific model characteristics, potentially related to debiasing.
Potential Use Cases
This model is suitable for applications where a smaller footprint and efficient inference are critical, while potentially benefiting from the characteristics introduced by the debiased model merge. It can be considered for tasks requiring a balance between performance and computational resources, leveraging its 32768 token context for handling longer inputs.