Overview

CatMacaroni14 is a 7 billion parameter language model developed by cookinai. It is the result of a slerp merge operation combining two base models: shadowml/Marcoro14-7B-slerp and rishiraj/CatPPT. The Marcoro14-7B-slerp component itself incorporates elements from EmbeddedLLM/Mistral-7B-Merge-14-v0.1, suggesting a lineage that includes Mistral-based architectures.

Merge Details

The model was created using a specific mergekit configuration, applying a slerp merge method. The merge parameters involved differential weighting for self_attn and mlp layers, with a fallback value for other tensors, indicating a fine-tuned approach to combining the strengths of its parent models. The merge was performed with bfloat16 precision.

Considerations

The creator notes potential contamination concerns regarding AIDC-ai-business/Marcoroni-7B-v3, a model that influenced one of CatMacaroni14's base components. Users are encouraged to report any findings to the HuggingFaceH4/open_llm_leaderboard to maintain data integrity.