andyx10/Qwen2.5-1.5B-Instruct-NLA-L18-av
The dormantx/Qwen2.5-1.5B-Instruct-NLA-L18-av is a 1.5 billion parameter model, part of a Natural Language Autoencoder (NLA) system, derived from Qwen/Qwen2.5-1.5B-Instruct. This specific 'AV' (Activation Verbalizer) component is designed to generate free-text descriptions of what a given activation from the base model's layer 18 residual stream encodes. It functions by injecting an L2-normalized activation into a fixed prompt, enabling interpretability research by verbalizing internal model states.
Loading preview...
Model Overview
The dormantx/Qwen2.5-1.5B-Instruct-NLA-L18-av is a 1.5 billion parameter model, serving as the "Activation Verbalizer" (AV) component of a Natural Language Autoencoder (NLA) system. It is built upon the Qwen/Qwen2.5-1.5B-Instruct architecture, specifically targeting the layer-18 residual stream.
Key Capabilities
- Activation Verbalization: This model's primary function is to take a raw, L2-normalized activation from the 18th hidden layer of the base
Qwen2.5-1.5B-Instructmodel and generate a natural language description of what that activation represents or encodes. - Interpretability Research: It facilitates understanding the internal workings of large language models by providing human-readable explanations for specific neural activations.
- Integrated Weights: The model comes with full merged weights, meaning LoRA adapters are already folded in, allowing for direct loading with
AutoModelForCausalLMwithout needing PEFT. - Paired Component: It is designed to work in conjunction with its sibling model,
dormantx/Qwen2.5-1.5B-Instruct-NLA-L18-ar, which handles the "Activation Reconstruction" aspect of the NLA.
Usage and Technical Details
Activations are injected into a fixed prompt using a special <|image_pad|> token. The model then generates a description based on this injected activation. This approach is valuable for researchers exploring mechanistic interpretability and understanding how LLMs process information at a granular level.