andyx10/Qwen2.5-1.5B-Instruct-NLA-L18-ar
The andyx10/Qwen2.5-1.5B-Instruct-NLA-L18-ar model is an activation reconstructor for the Qwen2.5-1.5B-Instruct language model. Developed by andyx10, it is part of a Natural Language Autoencoder (NLA) system, specifically designed to reconstruct activations from the layer-18 residual stream of the base Qwen model. This specialized component is crucial for interpretability research, allowing for the analysis and reconstruction of internal model states.
Loading preview...
Model Overview
This model, andyx10/Qwen2.5-1.5B-Instruct-NLA-L18-ar, functions as an activation reconstructor within a Natural Language Autoencoder (NLA) framework. It is specifically trained on the internal representations of the Qwen/Qwen2.5-1.5B-Instruct model, targeting the layer-18 residual stream.
Key Capabilities
- Activation Reconstruction: Designed to reconstruct the activations from a specific layer (layer 18) of the Qwen2.5-1.5B-Instruct model.
- Interpretability Research: Serves as a component for understanding and analyzing the internal workings of large language models, as described in the Natural Language Autoencoders paper.
- Integration with Verbalizer: Intended to be used in conjunction with its corresponding activation verbalizer,
dormantx/Qwen2.5-1.5B-Instruct-NLA-L18-av, to provide a complete NLA system.
Usage
The model can be loaded and used with transformers and safetensors to process tokenized inputs and extract reconstructed activation directions. The provided example demonstrates how to obtain the pooled reconstructed direction from the model's hidden states.
Good For
- Researchers and practitioners interested in mechanistic interpretability of large language models.
- Analyzing the internal representations and feature activations of Qwen2.5-1.5B-Instruct.
- Developing and experimenting with Natural Language Autoencoders.