ArchiveStudio/Phi-3.5-MoE-instruct
ArchiveStudio/Phi-3.5-MoE-instruct is a 41.9 billion parameter Mixture-of-Experts (MoE) decoder-only Transformer model developed by Microsoft, featuring 6.6 billion active parameters and a 128K token context length. Built upon Phi-3 datasets, it focuses on high-quality, reasoning-dense data, excelling in strong reasoning tasks, particularly code, math, and logic. The model is instruction-tuned and supports multilingual applications, demonstrating competitive performance against larger models in various benchmarks.
Loading preview...
ArchiveStudio/Phi-3.5-MoE-instruct: A Compact, High-Reasoning LLM
ArchiveStudio/Phi-3.5-MoE-instruct is a 41.9 billion parameter Mixture-of-Experts (MoE) model from Microsoft, utilizing 6.6 billion active parameters and supporting an extensive 128K token context length. It is built on high-quality, reasoning-dense synthetic and filtered public datasets, similar to the Phi-3 series, and has undergone rigorous fine-tuning for precise instruction adherence and safety.
Key Capabilities & Features
- Efficient Reasoning: Achieves strong reasoning capabilities, particularly in code, math, and logic, often outperforming larger models despite its smaller active parameter count.
- Multilingual Support: Designed for commercial and research use across multiple languages, showing competitive performance in multilingual benchmarks.
- Extended Context Window: Supports a 128K token context length, enabling long document summarization, QA, and multilingual context retrieval.
- Instruction-Tuned: Optimized through supervised fine-tuning, proximal policy optimization, and direct preference optimization for robust instruction following.
- MoE Architecture: Leverages a Mixture-of-Experts design (16x3.8B parameters with 2 experts active) for efficient performance.
Ideal Use Cases
- Resource-Constrained Environments: Suitable for memory/compute-constrained settings due to its efficient active parameter count.
- Latency-Sensitive Applications: Designed for scenarios where low latency is critical.
- Strong Reasoning Tasks: Excels in applications requiring robust logical, mathematical, and coding reasoning.
- Multilingual Applications: Effective for general-purpose AI systems and applications in various languages.
- Long Context Processing: Capable of handling and understanding extensive documents for tasks like summarization and question answering.