haoranli-ml/Llama-3-8B-RoPE-64k-Instruct

haoranli-ml/Llama-3-8B-RoPE-64k-Instruct Overview

This model is an instruction-tuned version of Llama-3-8B, featuring a significant enhancement through CoPE (Clipped RoPE). CoPE is a novel, plug-and-play modification to the standard RoPE (Rotary Positional Embedding) mechanism, designed to improve the model's performance and stability, particularly in long-context scenarios.

Key Capabilities & Innovations

• Enhanced Long-Context Handling: CoPE softly clips unstable low-frequency components within RoPE, leading to consistent performance gains both within the original training context window and during extrapolation to much longer contexts.
• Outlier Elimination: It effectively addresses and eliminates severe out-of-distribution (OOD) outliers, which are typically caused by periods exceeding the pre-training context window and are a primary source of instability during OOD extrapolation.
• Refined Semantic Signals: The enhancement refines long-range semantic signals by mitigating the inherent long-term decay of semantic attention introduced by the original RoPE.
• Prevention of Spectral Leakage: CoPE prevents spectral leakage that can arise from hard frequency truncation, which otherwise leads to oscillatory ringing in attention scores and introduces spurious correlations across relative token distances.

Good For

• Applications requiring robust performance with extended context lengths.
• Tasks where semantic understanding over long sequences is critical.
• Scenarios demanding stable and reliable extrapolation beyond the original training context window.