yamatazen/Ayla-Light-12B
yamatazen/Ayla-Light-12B is a 12 billion parameter language model created by yamatazen, merged using the SLERP method from nbeerbower/mistral-nemo-gutenberg-12B-v4 and Elizezen/Himeyuri-v0.1-12B. This model leverages a 32768 token context length and is designed to combine the strengths of its constituent models. Its primary application is general language understanding and generation tasks, benefiting from the diverse training data of its merged components.
Loading preview...
Model Overview
yamatazen/Ayla-Light-12B is a 12 billion parameter language model developed by yamatazen, created through a merge of existing pre-trained models. This model utilizes the SLERP (Spherical Linear Interpolation) merge method to combine the capabilities of its base components.
Merge Details
The Ayla-Light-12B model was constructed by merging two primary sources:
- nbeerbower/mistral-nemo-gutenberg-12B-v4 combined with nbeerbower/Mistral-Nemo-12B-abliterated-LORA
- Elizezen/Himeyuri-v0.1-12B
The merge configuration used a bfloat16 data type and specific t parameters for the SLERP method, aiming to balance the contributions of each constituent model. This approach allows Ayla-Light-12B to inherit and potentially enhance the strengths present in its merged predecessors.
Key Characteristics
- Parameter Count: 12 billion parameters.
- Context Length: Supports a substantial context window of 32768 tokens.
- Merge Method: Employs the SLERP technique for model combination, which is known for smoothly interpolating between model weights.
Potential Use Cases
Given its merged nature and 12B parameter size, Ayla-Light-12B is suitable for a range of general-purpose natural language processing tasks, including text generation, summarization, and question answering, benefiting from the diverse knowledge encoded in its base models.