https://github.com/openai/transformer-debugger Skip to content Toggle navigation Sign in * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Resources + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} openai / transformer-debugger Public * Notifications * Fork 75 * Star 1.8k * License MIT license 1.8k stars 75 forks Branches Tags Activity Star Notifications * Code * Pull requests 1 * Actions * Security * Insights Additional navigation options * Code * Pull requests * Actions * Security * Insights openai/transformer-debugger This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main BranchesTags Go to file Code Folders and files Last commit Last Name Name message commit date Latest commit History 9 Commits neuron_explainer neuron_explainer neuron_viewer neuron_viewer .gitignore .gitignore .isort.cfg .isort.cfg .pre-commit-config.yaml .pre-commit-config.yaml LICENSE LICENSE README.md README.md datasets.md datasets.md mypy.ini mypy.ini pyproject.toml pyproject.toml pytest.ini pytest.ini setup.py setup.py terminology.md terminology.md View all files Repository files navigation * README * MIT license Transformer Debugger Transformer Debugger (TDB) is a tool developed by OpenAI's Superalignment team with the goal of supporting investigations into specific behaviors of small language models. The tool combines automated interpretability techniques with sparse autoencoders. TDB enables rapid exploration before needing to write code, with the ability to intervene in the forward pass and see how it affects a particular behavior. It can be used to answer questions like, "Why does the model output token A instead of token B for this prompt?" or "Why does attention head H attend to token T for this prompt?" It does so by identifying specific components (neurons, attention heads, autoencoder latents) that contribute to the behavior, showing automatically generated explanations of what causes those components to activate most strongly, and tracing connections between components to help discover circuits. These videos give an overview of TDB and show how it can be used to investigate indirect object identification in GPT-2 small: * Introduction * Neuron viewer pages * Example: Investigating name mover heads, part 1 * Example: Investigating name mover heads, part 2 What's in the release? * Neuron viewer: A React app that hosts TDB as well as pages with information about individual model components (MLP neurons, attention heads and autoencoder latents for both). * Activation server: A backend server that performs inference on a subject model to provide data for TDB. It also reads and serves data from public Azure buckets. * Models: A simple inference library for GPT-2 models and their autoencoders, with hooks to grab activations. * Collated activation datasets: top-activating dataset examples for MLP neurons, attention heads and autoencoder latents. Setup Follow these steps to install the repo. You'll first need python/pip, as well as node/npm. Though optional, we recommend you use a virtual environment or equivalent: # If you're already in a venv, deactivate it. deactivate # Create a new venv. python -m venv ~/.virtualenvs/transformer-debugger # Activate the new venv. source ~/.virtualenvs/transformer-debugger/bin/activate Once your environment is set up, follow the following steps: git clone git@github.com:openai/transformer-debugger.git cd transformer-debugger # Install neuron_explainer pip install -e . # Set up the pre-commit hooks. pre-commit install # Install neuron_viewer. cd neuron_viewer npm install cd .. To run the TDB app, you'll then need to follow the instructions to set up the activation server backend and neuron viewer frontend. Making changes To validate changes: * Run pytest * Run mypy --config=mypy.ini . * Run activation server and neuron viewer and confirm that basic functionality like TDB and neuron viewer pages is still working Links * Terminology How to cite Please cite as: Mossing, et al., "Transformer Debugger", GitHub, 2024. BibTex citation: @misc{mossing2024tdb, title={Transformer Debugger}, author={Mossing, Dan and Bills, Steven and Tillman, Henk and Dupre la Tour, Tom and Cammarata, Nick and Gao, Leo and Achiam, Joshua and Yeh, Catherine and Leike, Jan and Wu, Jeff and Saunders, William}, year={2024}, publisher={GitHub}, howpublished={\url{https://github.com/openai/transformer-debugger}}, } About No description, website, or topics provided. Resources Readme License MIT license Activity Custom properties Stars 1.8k stars Watchers 13 watching Forks 75 forks Report repository Releases No releases published Packages 0 No packages published Contributors 6 * * * * * * Languages * Python 81.0% * TypeScript 18.9% * Other 0.1% Footer (c) 2024 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact * Manage cookies * Do not share my personal information You can't perform that action at this time.