wandb/pruned_mistral
The wandb/pruned_mistral is a 7 billion parameter Mistral-based language model, created by wandb, that has undergone pruning to reduce its size while maintaining performance. It was re-aligned using the Zephyr Recipe, incorporating both Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) training stages. This model is specifically designed for efficiency, offering a smaller footprint derived from selecting specific layers of the original Mistral model. Its primary strength lies in providing a more compact yet capable alternative for applications requiring Mistral-level performance with reduced computational overhead.
Loading preview...
Overview
The wandb/pruned_mistral is a 7 billion parameter language model based on the Mistral architecture. It distinguishes itself by being a pruned version of the original Mistral model, specifically re-aligned using the Zephyr Recipe. This approach aims to create a more efficient model by reducing its size while preserving key capabilities.
Key Characteristics
- Pruned Architecture: The model is constructed by selecting specific layers from the larger Mistral model, resulting in a smaller parameter count (7B).
- Two-Stage Training: It underwent a rigorous re-alignment process involving two distinct training stages:
- Supervised Fine-Tuning (SFT): Initial fine-tuning on a curated dataset.
- Direct Preference Optimization (DPO): Further optimization based on human preferences to enhance its conversational and instruction-following abilities.
- Zephyr Recipe Alignment: The re-alignment process follows the methodology outlined in the Zephyr Recipe, known for producing highly capable instruction-tuned models.
Potential Use Cases
This pruned Mistral model is particularly well-suited for scenarios where:
- Resource Efficiency is Crucial: Its smaller size makes it ideal for deployment on devices or environments with limited computational resources.
- Mistral-level Performance is Desired: Users need the general capabilities of a Mistral-based model but in a more compact form factor.
- Instruction Following: The SFT and DPO alignment suggest strong performance in instruction-following tasks and conversational AI.
For more technical details and the code used for pruning, refer to the associated GitHub repository: github.com/tcapelle/shear. The development process can also be explored in the W&B workspace: wandb.ai/llm_surgery/shearllama/.