Overview

rubenroy/Zurich-14B-GCv2-5m is a 14.7 billion parameter causal language model developed by Ruben Roy. It is a fine-tuned version of Alibaba's Qwen 2.5 14B Instruct model, leveraging its Transformer architecture which includes RoPE, SwiGLU, RMSNorm, and Attention QKV bias, with 48 layers and 40 attention heads for Q and 8 for KV.

Key Differentiator: GammaCorpus v2-5m Fine-tuning

This model's primary distinction is its fine-tuning on the GammaCorpus v2-5m dataset. GammaCorpus is a collection of structured and filtered multi-turn conversations, designed to enhance conversational capabilities. The fine-tuning process involved 60 epochs over approximately 90 minutes using an A100 GPU and the Unsloth framework.

Intended Use

Zurich-14B-GCv2-5m is designed to excel in conversational AI tasks, particularly those benefiting from structured, multi-turn dialogue. Its fine-tuning on GammaCorpus aims to provide robust performance in generating coherent and contextually relevant responses in interactive scenarios. Users should be aware of potential biases, as efforts to mitigate them are ongoing.

Licensing

The model is released under the Apache 2.0 License.