Hugging Face Progenitor-V1.1-LLaMa-70B: A Merged Llama-Based Model
Progenitor-V1.1-LLaMa-70B is a 70 billion parameter language model developed by Tarek07, resulting from a series of experiments in merging Llama-based models. This model was constructed using the della_linear merge method, building upon nbeerbower/Llama-3.1-Nemotron-lorablated-70B as its foundational base.
Merge Composition
The model integrates contributions from five distinct Llama-based models, each weighted at 20% with a density of 0.7 during the merge process. The constituent models include:
EVA-UNIT-01/EVA-LLaMA-3.33-70B-v0.1Sao10K/L3.1-70B-Hanami-x1Sao10K/70B-L3.3-Cirrus-x1TheDrummer/Anubis-70B-v1SicariusSicariiStuff/Negative_LLAMA_70B
This aggressive merging strategy aims to combine the diverse capabilities and characteristics of these individual models into a unified, high-performance language model. The merge configuration utilized specific epsilon and lambda parameters, with bfloat16 dtype and base tokenizer source, indicating a focus on robust integration of the merged components.