Labradorite-13b: A LLaMA-2 Derivative by IBM Research

Labradorite-13b is a 13 billion parameter language model developed by IBM Research, built upon the LLaMA-2-13b architecture. Its key differentiator is the Large-scale Alignment for chatBots (LAB) methodology, a novel synthetic data-based alignment tuning approach. This method allows the model to incrementally acquire new knowledge and skills without suffering from catastrophic forgetting, a common challenge in LLM training.

Key Capabilities & Methodology

The LAB methodology involves three core components:

• Taxonomy-driven data curation: Utilizes a hierarchical taxonomy of seed examples to guide the generation of diverse synthetic data, ensuring broad coverage of knowledge domains and skills.
• Large-scale synthetic data generation: Employs Mixtral-8x7B-Instruct as a teacher model, sampling local examples within the taxonomy to prompt generation, which allows it to compete with models trained on GPT-4 generated data.
• Two-phased training with replay buffers: Consists of distinct knowledge tuning (simple then complicated knowledge) and skills tuning phases, incorporating replay buffers to prevent forgetting.

Performance Highlights

Labradorite-13b demonstrates competitive performance against other LLaMA-2-13b derivatives like Orca-2 and WizardLM-13B-V1.2. On the MTBench (Avg), it scores 7.23, and on MMLU (5-shot), it achieves 58.89. It also shows strong results on ARC-C (61.69), HellaSwag (83.15), and Winogrande (79.56).

Recommended Use Cases

This model is well-suited for general chatbot applications, particularly where a robust, incrementally trainable model derived from LLaMA-2 is desired. Its alignment process focuses on creating a cautious, helpful, and harmless assistant, making it a strong candidate for conversational AI systems.