kitft/nla-gemma3-27b-L41-ar
The kitft/nla-gemma3-27b-L41-ar model is a 27 billion parameter activation reconstructor (AR) fine-tuned from Google's Gemma 3 architecture, designed for interpretability research. This model translates natural language descriptions back into hidden-state vectors, forming one half of a Natural Language Autoencoder (NLA) pair. It is specifically repurposed for activation decoding, not general-purpose language generation, and works in conjunction with its activation verbalizer counterpart. The model was fine-tuned on a dataset combining WildChat-1M and Ultra-FineWeb, with an in-distribution fve_nrm of 0.763.
Loading preview...
Model Overview
The kitft/nla-gemma3-27b-L41-ar is a 27 billion parameter model, specifically the Activation Reconstructor (AR) component of a Natural Language Autoencoder (NLA) pair. Fine-tuned from google/gemma-3-27b-it, its primary function is to map natural language descriptions back to hidden-state vectors within an LLM's residual stream. It is designed to be used alongside its counterpart, kitft/nla-gemma3-27b-L41-av, which performs the inverse operation (vector to description).
Key Capabilities & Purpose
- LLM Interpretability: This model is an advanced tool for understanding the internal workings of large language models. It allows researchers to "read out" what a residual-stream activation signifies by converting a natural language description into a corresponding activation vector.
- Activation Decoding: Its core utility lies in decoding and reconstructing activations, enabling the measurement of how effectively a natural language description captures the meaning of a specific activation.
- NLA Framework: It is an integral part of the Natural Language Autoencoder framework, which provides unsupervised explanations of LLM activations, as detailed in the associated research paper.
Important Considerations
- Not a General-Purpose LLM: This model has been entirely repurposed for activation decoding and is not suitable for general-purpose language generation, chat, or instruction following. Its fine-tuning has optimized it specifically for interpretability tasks.
- Paired Usage: It is intended to be used in conjunction with its activation verbalizer (AV) counterpart for full NLA functionality.
- Training Data: Fine-tuned on a blend of WildChat-1M and Ultra-FineWeb datasets.
Usage
Detailed usage instructions, including SGLang launch, NLAClient/NLACritic, and embedding-injection specifics, are available in the nla-inference README.