toughtty/phi-2
toughtty/phi-2 is a 2.7 billion parameter Transformer-based causal language model developed by Microsoft. Trained on a combination of synthetic NLP texts and filtered web data, it demonstrates strong performance in common sense, language understanding, and logical reasoning among models under 13 billion parameters. This model is primarily intended for research into safety challenges and excels in QA, chat, and code generation formats.
Loading preview...
Model Overview
toughtty/phi-2 is a 2.7 billion parameter Transformer model developed by Microsoft, building upon the data sources of Phi-1.5 and augmented with new synthetic NLP texts and filtered web content. It achieves near state-of-the-art performance in benchmarks for common sense, language understanding, and logical reasoning among models with fewer than 13 billion parameters.
Key Capabilities & Intended Uses
- Research into Safety Challenges: Phi-2 is released as an open-source model to facilitate research into critical safety aspects such as reducing toxicity, understanding societal biases, and enhancing controllability.
- QA Format: Well-suited for question-answering tasks, supporting both standalone questions and an "Instruct: \nOutput:" format for concise answers.
- Chat Format: Capable of engaging in multi-turn conversational exchanges.
- Code Format: Effective for generating code, particularly in Python, with common packages like
typing,math,random,collections,datetime, anditertools.
Limitations
- Inaccurate Code and Facts: May generate incorrect code or factual statements; outputs should be treated as starting points.
- Limited Code Scope: Primarily trained on Python with specific packages; users should verify code using other languages or less common packages.
- Unreliable Instruction Adherence: Not instruction fine-tuned, so it may struggle with complex or nuanced instructions.
- Language Limitations: Primarily understands standard English; informal English, slang, or other languages may lead to misinterpretations.
- Societal Biases & Toxicity: Despite training efforts, the model may still reflect societal biases or produce harmful content if explicitly prompted.
- Verbosity: As a base model trained on textbook-like data, it can sometimes produce irrelevant or extra text.
Training Details
Phi-2 was trained on 1.4 trillion tokens over 14 days using 96xA100-80G GPUs. The dataset size was 250 billion tokens, combining synthetic NLP data and filtered web data from Falcon RefinedWeb and SlimPajama.