The IntelLabs/sqft-phi-3-mini-4k-30-base is a 4 billion parameter sparse language model developed by IntelLabs, derived from Microsoft's Phi-3-mini-4k-instruct. It utilizes the Wanda sparse method to achieve 30% sparsity, focusing on efficient model adaptation. This base model is designed for research into low-cost model adaptation within sparse foundation models.
Loading preview...
SQFT Phi-3-mini-4k-30-base Overview
This model, developed by IntelLabs, is a 4 billion parameter sparse variant of Microsoft's Phi-3-mini-4k-instruct. It incorporates the Wanda sparse method to achieve a 30% sparsity level, aiming for efficient and low-cost model adaptation. The development is part of ongoing research into hardware-aware automated machine learning and sparse foundation models.
Key Characteristics
- Base Model: Derived from
microsoft/Phi-3-mini-4k-instruct. - Sparsity: Achieves 30% sparsity using the Wanda method.
- Parameter Count: 4 billion parameters.
- Context Length: Supports a 4k (4096 token) context window.
- Quantization: This specific base model does not include quantization.
Research Focus
This model is primarily a research artifact, detailed in papers such as "SQFT: Low-cost Model Adaptation in Low-precision Sparse Foundation Models" and "Low-Rank Adapters Meet Neural Architecture Search for LLM Compression." It serves as a foundation for exploring efficient model compression and adaptation techniques, particularly for scenarios requiring reduced computational resources.