sthenno/tempesthenno-ms-0314-001
sthenno/tempesthenno-ms-0314-001 is a 14.8 billion parameter language model created by sthenno, developed using the Model Stock merge method. This model integrates several 'tempesthenno' and 'tempestissimo' components, with 'sthenno-com/miscii-14b-0218' serving as its base. It is designed for general language tasks, leveraging its merged architecture to combine diverse capabilities from its constituent models.
Loading preview...
Model Overview
sthenno/tempesthenno-ms-0314-001 is a 14.8 billion parameter language model developed by sthenno. It was created using the Model Stock merge method, a technique described in the paper "Model Stock: A Method for Merging Pre-trained Language Models." This approach combines the strengths of multiple pre-trained models into a single, more capable entity.
Merge Details
The model's foundation is sthenno-com/miscii-14b-0218, which served as the base model for the merge. Several other models were integrated, including:
sthenno/tempestissimo-14b-0309sthenno/tempesthenno-sft-0309-ckpt10- Additional internal 'tempesthenno-sft' checkpoints (
stage1-ckpt50,stage3-ckpt30,stage1-ckpt100)
This merging strategy aims to leverage the distinct characteristics and training of each component model to enhance overall performance. The configuration used for the merge specified bfloat16 as the data type and applied normalization during parameter integration.