Anthropic's published research demystifies some aspects of AI's algorithmic behavior

The world of artificial intelligence is often described as a “black box” due to its opaque and complex nature. Now, Anthropic, one of the industry’s leading companies, has published research aimed at demystifying some aspects of AI’s algorithmic behavior.

Exploring AI Decisions

Anthropic’s research focuses on understanding why their AI chatbot, Claude, favors certain subjects over others. AI systems, designed to mirror the human brain’s structure, intake and process data to make decisions or predictions. These systems are trained on vast sets of data, enabling them to establish algorithmic connections.

However, humans observers often struggle to understand how the AI system reached a particular output, leading to the emergence of AI interpretation. This field attempts to trace the machine’s decision-making path to better comprehend its output.

Deciphering Neural Networks

In AI interpretation, a “feature” equates to a pattern of activated “neurons” within a neural net, essentially a concept that the algorithm refers back to. Understanding more features can help researchers comprehend how specific inputs influence the neural net to generate particular outputs.

Anthropic researchers have utilized a process known as “dictionary learning” to identify which parts of Claude’s neural network correspond to specific concepts. This approach has provided insights into the model’s ‘reasoning’ for a given response.

Case Study: The Golden Gate Bridge Feature

The research team has shared an interesting example of their findings. They identified a feature associated with the Golden Gate Bridge, defining a set of neurons that, when fired together, indicated Claude was “thinking” about the San Francisco landmark. Moreover, when similar neuron sets fired, they evoked related subjects such as Alcatraz, California Governor Gavin Newsom, and the movie Vertigo, set in San Francisco.

Through this process, the team identified millions of features, providing a kind of Rosetta Stone to decode Claude’s neural net.

Transparency in AI Research

While for-profit companies like Anthropic may have business-related motivations for their research, the team’s study is public, allowing anyone to read it and draw their own conclusions about their findings and methodologies. This openness contributes to the ongoing effort to demystify AI, shedding light on the complex workings of this cutting-edge technology.