Archangel SFT+KTO Llama 13B Overview

ContextualAI's archangel_sft-kto_llama13b is a 13 billion parameter model built on the Llama architecture. It distinguishes itself through its unique optimization strategy, combining Supervised Fine-Tuning (SFT) with Kahneman-Tversky Optimization (KTO) loss functions. This approach aims to enhance model alignment and performance.

Key Capabilities

• Advanced Alignment: The model has been aligned using a diverse set of human preference datasets, including SHP, Anthropic HH, and Open Assistant, contributing to more human-like and helpful responses.
• TuluV2 Prompt Format: It is designed to be prompted using the TuluV2 format, which clearly delineates user and assistant turns, facilitating structured conversations.
• Conditional SFT Control Tokens: Models trained with conditional SFT include special tokens like <|good|> and <|bad|> in their embeddings, allowing for fine-grained control over generation by appending these to the prompt.

Good For

• Conversational AI: Its alignment on human preference datasets makes it well-suited for dialogue systems and interactive applications.
• Research into Alignment Techniques: The model serves as an example of applying SFT+KTO loss for alignment, offering insights for researchers in the field of LLM optimization. Further details on the methodology are available in the code repository and technical paper.