dphn/dolphin-llama-13b

Overview

dphn/dolphin-llama-13b is a 13 billion parameter language model built upon the Llama-1 architecture. It was developed by Eric Hartford and his team, focusing on creating an uncensored model by filtering out alignment, refusal, avoidance, and bias from its training data. This design choice ensures the model is highly compliant to user requests, providing a neutral base for developers to integrate their own alignment or safety layers.

Training and Dataset

The model's training utilized an open-source implementation of Microsoft's Orca dataset, which includes augmented FLANv2 instructions with GPT-4 and GPT-3.5 completions. The dataset underwent extensive cleaning, deduplication, and filtering to remove inherent biases. Training involved separate epochs on the FLAN-1m (GPT-4 completions) and FLAN-5m (GPT-3.5 completions) datasets, totaling approximately 600 hours on 8x H100 GPUs. The prompt format is similar to Vicuna, incorporating a SYSTEM: field.

Key Characteristics

• Uncensored Nature: Designed to be highly compliant with all requests, including potentially unethical ones, requiring users to implement their own alignment. This allows for maximum flexibility in application.
• Orca-based Fine-tuning: Leverages a cleaned and filtered version of the Orca dataset, known for its complex explanation traces.
• Llama-1 Architecture: Built on the Llama-1 foundation, making it suitable for non-commercial use.

Performance (Open LLM Leaderboard)

While evaluation is ongoing, initial results from the Open LLM Leaderboard indicate a competitive performance for its size class:

• Avg.: 48.6
• ARC (25-shot): 55.55
• HellaSwag (10-shot): 77.11
• MMLU (5-shot): 52.16

Use Cases

This model is ideal for developers and researchers who require a highly flexible and uncensored base model. It is particularly useful for applications where custom alignment, bias control, or specific ethical guidelines need to be implemented by the user, rather than relying on pre-baked model behaviors.