ArchiveStudio/phi-2

Model Overview

ArchiveStudio/phi-2 is a 2.7 billion parameter Transformer model developed by Microsoft. It was trained on a unique blend of data sources, including synthetic NLP texts and filtered web content, building upon the methodology of its predecessor, Phi-1.5. Phi-2 achieves near state-of-the-art performance in benchmarks for common sense, language understanding, and logical reasoning among models with fewer than 13 billion parameters.

Key Capabilities & Features

• Architecture: Transformer-based with a next-word prediction objective.
• Parameter Count: 2.7 billion parameters.
• Context Length: Supports a context length of 2048 tokens.
• Training Data: Trained on 1.4 trillion tokens, including synthetic NLP data generated by AOAI GPT-3.5 and filtered web data from Falcon RefinedWeb and SlimPajama, assessed by AOAI GPT-4.
• Research Focus: Released to provide the research community with a small, non-restricted model to explore critical safety challenges like toxicity reduction, bias understanding, and controllability.

Intended Uses

Phi-2 is specifically designed and best suited for the following prompt formats:

• QA Format: Responding to standalone questions or structured "Instruct: \nOutput:" queries.
• Chat Format: Engaging in multi-turn conversational exchanges.
• Code Format: Generating code snippets, particularly in Python, for common packages.

Limitations

• May generate inaccurate code and facts; outputs should be treated as starting points.
• Primarily trained on Python with common packages; other languages or less common packages may require manual verification.
• Has not undergone instruction fine-tuning, which may lead to struggles with complex instructions.
• Primarily understands standard English; informal English or other languages may pose challenges.
• May exhibit societal biases and can produce harmful content if explicitly prompted.
• Can be verbose, often producing extra text due to its textbook-like training data.