SQFT Phi-3-mini-4k-30-base Overview

This model, developed by IntelLabs, is a 4 billion parameter sparse variant of Microsoft's Phi-3-mini-4k-instruct. It incorporates the Wanda sparse method to achieve a 30% sparsity level, aiming for efficient and low-cost model adaptation. The development is part of ongoing research into hardware-aware automated machine learning and sparse foundation models.

Key Characteristics

• Base Model: Derived from microsoft/Phi-3-mini-4k-instruct.
• Sparsity: Achieves 30% sparsity using the Wanda method.
• Parameter Count: 4 billion parameters.
• Context Length: Supports a 4k (4096 token) context window.
• Quantization: This specific base model does not include quantization.

Research Focus

This model is primarily a research artifact, detailed in papers such as "SQFT: Low-cost Model Adaptation in Low-precision Sparse Foundation Models" and "Low-Rank Adapters Meet Neural Architecture Search for LLM Compression." It serves as a foundation for exploring efficient model compression and adaptation techniques, particularly for scenarios requiring reduced computational resources.