jesusvilela/manifoldgl
ManifoldGL by jesusvilela is a parameter-efficient adapter that applies hyperbolic geometry to the latent space of large language models. It treats token meaning as a fiber over a hyperbolic base manifold, enhancing the representation of hierarchical structures. Fine-tuned on Qwen2.5-7B, this adapter significantly improves task accuracy on the ARC-AGI benchmark, demonstrating a 131.5% relative improvement. It is designed to be loaded with PEFT to enhance reasoning capabilities by leveraging hyperbolic embeddings.
Loading preview...
ManifoldGL: Hyperbolic Geometry for LLMs
ManifoldGL is a unique parameter-efficient adapter developed by jesusvilela that introduces hyperbolic geometry into the latent space of large language models. Unlike traditional Euclidean embeddings, which struggle with hierarchical data, hyperbolic space excels at representing complex, hierarchical relationships due to its exponential growth capacity.
Key Capabilities and Innovations
- Hyperbolic Latent Space: Models token meaning as a fiber over a hyperbolic base manifold (Poincaré ball), where attentions are computed using geodesic distance.
- Enhanced Hierarchical Representation: Leverages the exponential growth of hyperbolic space to better preserve local and global relationships in hierarchical structures, outperforming Euclidean embeddings in tasks like lexical entailment.
- Semantic Structure Preservation: Employs a sheaf-theoretic consistency loss and natural gradient optimization during training to maintain semantic integrity.
- Significant ARC-AGI Improvement: When fine-tuned on Qwen2.5-7B, ManifoldGL boosts ARC-AGI benchmark accuracy from 12.4% to 28.7%, a 131.5% relative improvement.
- High Manifold Faithfulness: Achieves a Manifold Faithfulness Rate (MFR) of 94.2%, indicating strong adherence to hyperbolic constraints.
When to Use This Model
This adapter is ideal for researchers and developers looking to enhance the reasoning and hierarchical understanding capabilities of Qwen2.5-7B models. It is particularly beneficial for tasks requiring a nuanced grasp of semantic relationships and complex structures. The adapter can be easily integrated using the PEFT library, projecting latent states into hyperbolic space during generation. It is recommended to use FP32 precision for optimal stability.