Fischerboot/ll3-test-6
Fischerboot/ll3-test-6 is an 8 billion parameter language model created by Fischerboot, developed through a SLERP merge of the Fischerboot/ll3-test-5 model. This model leverages specific layer ranges and parameter weighting for self-attention and MLP blocks to refine its capabilities. It is designed for general language tasks, building upon its base model's characteristics.
Loading preview...
Model Overview
Fischerboot/ll3-test-6 is an 8 billion parameter language model developed by Fischerboot. This model was created using the mergekit tool, specifically employing the SLERP (Spherical Linear Interpolation) merge method.
Merge Details
The model is a merge of a single base model, Fischerboot/ll3-test-5. The merge process involved combining different layer ranges of the ll3-test-5 model, with specific parameter adjustments for self-attention and MLP blocks. This technique allows for fine-grained control over how the characteristics of the base model are integrated and potentially enhanced in the merged output.
Configuration
The merge configuration utilized distinct t values for self-attention and MLP filters, indicating a tailored approach to blending the model's components. The process was conducted using bfloat16 dtype, which is common for large language model training and merging due to its balance of precision and memory efficiency.
Intended Use
As a merged model, Fischerboot/ll3-test-6 is suitable for general natural language processing tasks, inheriting and potentially refining the capabilities of its base model, Fischerboot/ll3-test-5.