Fischerboot/13-test
Fischerboot/13-test is an 8 billion parameter language model created by Fischerboot, resulting from a merge of pre-trained models using the SLERP method. This model is a merge of Fischerboot/12-test, specifically combining different layer ranges and parameter values. It is designed for general language tasks, leveraging its merged architecture to potentially enhance performance over its base model.
Loading preview...
Model Overview
Fischerboot/13-test is an 8 billion parameter language model developed by Fischerboot. This model was created through a merge of pre-trained language models, specifically utilizing the mergekit tool.
Merge Details
The model was constructed using the SLERP merge method, which combines the weights of different models. The primary model included in this merge was Fischerboot/12-test.
Configuration
The merge process involved specific layer ranges from the source model:
- Layers 0 to 18 from Fischerboot/12-test
- Layers 1 to 19 from Fischerboot/12-test
Custom parameters were applied during the merge, with varying values for self-attention (self_attn) and multi-layer perceptron (mlp) filters, and a general parameter value of 0.5. The model was processed with bfloat16 data type. This merging approach aims to combine the strengths of the constituent model in a structured manner.