OwenArli/ArliAI-Llama-3-8B-Instruct-ORPO-v0.1

ArliAI-Llama-3-8B-Instruct-ORPO-v0.1 Overview

This model is an 8 billion parameter instruction-tuned variant of Meta-Llama-3-8B-Instruct, developed by OwenArli. It leverages the ORPO (Optimized Reward Policy Optimization) fine-tuning method to enhance its performance and behavioral characteristics.

Key Characteristics & Training

The fine-tuning process utilized a diverse set of datasets, including:

• Intel/orca_dpo_pairs
• argilla/distilabel-math-preference-dpo
• unalignment/toxic-dpo-v0.2
• M4-ai/prm_dpo_pairs_cleaned
• jondurbin/truthy-dpo-v0.1

Notably, the inclusion of "toxic" datasets was intended to reduce model refusals, resulting in a model that is still relatively safe but less prone to refusing prompts compared to the original Meta model. While ORPO fine-tuning has shown improvements in some metrics, the developer notes that it may reduce others.

Usage and Format

The model adheres to the standard Llama 3 instruction format, utilizing specific tokens for system, user, and assistant turns:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

{{ system_prompt }}<|eot_id|><|start_header_id|>user<|end_header_id|>

{{ user_message_1 }}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

{{ model_answer_1 }}<|eot_id|><|start_header_id|>user<|end_header_id|>

{{ user_message_2 }}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Ideal Use Cases

This model is particularly suited for applications where:

• A less restrictive response style is desired.
• The base Llama 3 8B Instruct model's refusal rate is too high.
• Tasks benefit from the ORPO fine-tuning approach, especially in areas covered by the training datasets like mathematical reasoning or general instruction following.