kmseong/llama3.1_8b_base-Safety-FT-lr3e-5

Model Overview

The kmseong/llama3.1_8b_base-Safety-FT-lr3e-5 is an 8 billion parameter language model built upon the Llama 3.1 base architecture, supporting a substantial context length of 32768 tokens. This model has undergone specific fine-tuning to enhance safety alignment.

Key Technical Details

• Architecture: Llama 3.1 base with 8 billion parameters.
• Context Length: Supports up to 32768 tokens.
• Training Methodology: The model's training incorporates attention mechanisms (q, k, v) and MLP (up, down) with perlayer application. A notable aspect is its subsequent non-freeze training phase.
• Safety Alignment: The model is explicitly fine-tuned for safety, leveraging a technique referred to as "Weight space Rotation Process" (Warp), as detailed in its associated citation.

Good For

• Applications requiring a Llama 3.1-based model with enhanced safety characteristics.
• Use cases where a large context window (32768 tokens) is beneficial.
• Research and development into safety alignment techniques for large language models.