Fischerboot/17-test

TEXT GENERATIONPricing:Input $0.37 / Cached $0.074 / Output $0.38Concurrent Unit Cost:1Model Size:8BQuant:FP8Context Size:8kTool Calling:SupportedPublished:Jul 6, 2024Architecture:Transformer Featherless Exclusive Cold

Fischerboot/17-test is an 8 billion parameter language model created by Fischerboot, resulting from a SLERP merge of layers from Fischerboot/16-test. This model is specifically designed as a merged iteration, focusing on combining specific layer ranges and parameter values from its base model. It is intended for use cases requiring a model derived from a precise, layer-wise merging strategy.

Loading preview...

Overview

Fischerboot/17-test is an 8 billion parameter language model developed by Fischerboot. It was created using the mergekit tool, specifically employing the SLERP (Spherical Linear Interpolation) merge method.

Merge Details

This model is a direct merge of layers from a single base model, Fischerboot/16-test. The merging process involved combining different layer ranges from the source model:

  • Layers 0 through 14 from Fischerboot/16-test.
  • Layers 1 through 15 from Fischerboot/16-test.

The SLERP method was applied with specific parameter configurations for self_attn and mlp components, as well as a general value of 0.5, indicating a weighted interpolation strategy. The model was processed using bfloat16 data type.

Key Characteristics

  • Architecture: Merged model derived from Fischerboot/16-test.
  • Merge Method: Utilizes the SLERP technique for combining model weights.
  • Parameter Count: 8 billion parameters.
  • Context Length: 8192 tokens.

Intended Use Cases

This model is suitable for developers and researchers interested in:

  • Experimenting with models created via advanced merging techniques like SLERP.
  • Evaluating the impact of specific layer-range merging on model performance.
  • Building upon a base model (Fischerboot/16-test) with a modified weight distribution.

Finetunes
2 models