IntelLabs/sqft-mistral-7b-v0.3-50-base
The IntelLabs/sqft-mistral-7b-v0.3-50-base model is a 7 billion parameter language model developed by IntelLabs, derived from Mistral-7B-v0.3. It incorporates a 50% sparsity using the Wanda pruning method, focusing on efficient model adaptation. This base model is designed for low-cost model adaptation within low-precision sparse foundation models, making it suitable for resource-constrained environments.
Loading preview...
Overview
The sqft-mistral-7b-v0.3-50-base is a 7 billion parameter language model developed by IntelLabs. It is built upon the mistralai/Mistral-7B-v0.3 architecture and features a 50% sparsity level achieved through the Wanda pruning method. This model is specifically designed for efficient deployment and adaptation in scenarios where computational resources are limited, as detailed in the associated research papers.
Key Characteristics
- Source Model: Derived from
mistralai/Mistral-7B-v0.3. - Sparsity: Achieves 50% sparsity using the Wanda method, which is a simple yet effective pruning approach.
- Quantization: The base model itself does not incorporate quantization.
- Research Focus: Developed as part of research into "Low-cost Model Adaptation in Low-precision Sparse Foundation Models" and "Low-Rank Adapters Meet Neural Architecture Search for LLM Compression."
Intended Use Cases
This model is particularly well-suited for:
- Resource-constrained environments: Its sparse nature allows for more efficient inference and deployment.
- Model adaptation: Designed to facilitate low-cost adaptation within sparse, low-precision foundation models.
- Research in model compression: Serves as a base for exploring sparse and quantized model architectures.