Fischerboot/18-test
Fischerboot/18-test is an 8 billion parameter language model created by Fischerboot, resulting from a merge of pre-trained models using the SLERP method. Specifically, it is a merge of Fischerboot/17-test with itself, applying different layer ranges and parameter interpolation values. This model is designed to explore the effects of specific layer-wise merging strategies on model performance and characteristics, offering a unique configuration for research and specialized applications.
Loading preview...
Model Overview
Fischerboot/18-test is an 8 billion parameter language model developed by Fischerboot. This model was created through a merge operation using the mergekit tool, specifically employing the SLERP (Spherical Linear Interpolation) merge method. It is derived from the Fischerboot/17-test model.
Merge Details
The model's unique configuration stems from merging Fischerboot/17-test with itself, but with distinct layer ranges and parameter weighting:
- Source 1: Layers 0 to 13 from
Fischerboot/17-test. - Source 2: Layers 1 to 14 from
Fischerboot/17-test.
The SLERP merge method was applied with specific interpolation values (t) for different components:
- Self-attention layers:
tvalues varied across layers (0, 0.5, 0.3, 0.7, 1). - MLP layers:
tvalues also varied (1, 0.5, 0.7, 0.3, 0). - Other parameters: A default
tvalue of 0.5 was used.
This intricate merging strategy aims to create a model with potentially altered or enhanced characteristics compared to its base model, Fischerboot/17-test, by selectively blending different parts of the same model. The model uses bfloat16 for its data type.
Potential Use Cases
- Research into model merging techniques: Ideal for studying the impact of layer-wise SLERP merging.
- Experimentation with model architectures: Provides a unique configuration for testing specific hypotheses about model composition.
- Specialized applications: May exhibit emergent properties suitable for niche tasks not covered by the base model.