ibm-research/labradorite-13b
Labradorite-13b is a 13 billion parameter LLaMA-2-13b derivative model developed by IBM Research, fine-tuned using their novel Large-scale Alignment for chatBots (LAB) methodology. This model leverages a taxonomy-driven synthetic data generation process with Mixtral-8x7B-Instruct as a teacher model, enabling incremental knowledge and skill acquisition without catastrophic forgetting. It demonstrates competitive performance on benchmarks like MTBench and MMLU, making it suitable for general-purpose chatbot applications.
Loading preview...
Labradorite-13b: A LLaMA-2 Derivative by IBM Research
Labradorite-13b is a 13 billion parameter language model developed by IBM Research, built upon the LLaMA-2-13b architecture. Its key differentiator is the Large-scale Alignment for chatBots (LAB) methodology, a novel synthetic data-based alignment tuning approach. This method allows the model to incrementally acquire new knowledge and skills without suffering from catastrophic forgetting, a common challenge in LLM training.
Key Capabilities & Methodology
The LAB methodology involves three core components:
- Taxonomy-driven data curation: Utilizes a hierarchical taxonomy of seed examples to guide the generation of diverse synthetic data, ensuring broad coverage of knowledge domains and skills.
- Large-scale synthetic data generation: Employs Mixtral-8x7B-Instruct as a teacher model, sampling local examples within the taxonomy to prompt generation, which allows it to compete with models trained on GPT-4 generated data.
- Two-phased training with replay buffers: Consists of distinct knowledge tuning (simple then complicated knowledge) and skills tuning phases, incorporating replay buffers to prevent forgetting.
Performance Highlights
Labradorite-13b demonstrates competitive performance against other LLaMA-2-13b derivatives like Orca-2 and WizardLM-13B-V1.2. On the MTBench (Avg), it scores 7.23, and on MMLU (5-shot), it achieves 58.89. It also shows strong results on ARC-C (61.69), HellaSwag (83.15), and Winogrande (79.56).
Recommended Use Cases
This model is well-suited for general chatbot applications, particularly where a robust, incrementally trainable model derived from LLaMA-2 is desired. Its alignment process focuses on creating a cautious, helpful, and harmless assistant, making it a strong candidate for conversational AI systems.