ManifoldGL: Hyperbolic Geometry for LLMs

ManifoldGL is a parameter-efficient adapter that introduces hyperbolic geometry into the latent space of large language models. Unlike traditional Euclidean embeddings, hyperbolic space excels at representing hierarchical structures, which are common in language semantics. This adapter models token meaning as a fiber over a hyperbolic base manifold (a Poincaré ball), where latent states are projected onto the ball and attention is computed using geodesic distance. This method aims to better preserve both local and global relationships within semantic hierarchies.

Key Capabilities & Features

• Hyperbolic Latent Space: Enforces hyperbolic geometry on token embeddings, allowing for more effective representation of hierarchical semantic structures.
• Improved Reasoning: Achieves a 131.5% relative improvement in task accuracy on the ARC-AGI benchmark (from 12.4% to 28.7%) when fine-tuned on Qwen2.5-7B.
• Manifold Faithfulness: Maintains a high Manifold Faithfulness Rate (MFR) of 94.2%, indicating strong adherence to hyperbolic constraints.
• Parameter-Efficient: Functions as an adapter, making it efficient to integrate with existing LLMs like Qwen2.5-7B using PEFT.

When to Use ManifoldGL

ManifoldGL is particularly beneficial for applications requiring enhanced reasoning and understanding of complex, hierarchical relationships in data. Its ability to model semantics in hyperbolic space makes it suitable for tasks where traditional Euclidean embeddings might fall short in capturing intricate semantic structures. It can be loaded with PEFT on compatible Qwen2.5-7B models, with FP32 precision recommended for stability.