Can We Understand How Large Language Models Reason?

Large language models can write essays, solve math problems, and generate computer code, but it’s not fully understood how they do it. Researchers can observe the billions of parameters inside these systems changing during training, yet the internal logic of the models remains largely hidden. In a sense, the engineering is ahead of the science. Can science catch up and make LLMs and other deep neural networks mechanistically interpretable?

Thomas Icard, a professor of philosophy and computer science at Stanford University, is contributing to this effort using tools from logic and cognitive science. He is a researcher in the growing field of mechanistic interpretability. “It’s striking how much progress there already has been in the last few years on basically every dimension,” Icard said, “from how much a model’s behaviors and representations reflect patterns in training data, and how post-training reshapes that behavior, to deep connections between internalized abstractions and generalization.”

One of the central questions in this field is whether neural networks merely mimic reasoning, or whether they internally build elements of a logical or algorithmic reasoning system. Icard’s main contribution has been to develop, along with his students and collaborators, a rigorous framework for investigating this question. “We have shown how tools from causality can make precise what it means for a neural network to implement an algorithm at a higher level of abstraction,” he explained. “Of course, a neural network implements algorithms at a very low level of abstraction, involving matrix multiplications and the like. But we are searching for concepts that are more intuitive and interpretable, such as those explored in cognitive science, linguistics, and logic.”

This search resembles the way physicists abstract from the statistical behavior of individual molecules to the ideal gas law, which describes a gas in terms of pressure and temperature rather than in terms of colliding particles. In the simplest analogy for neural networks, the activity of many individual neurons can similarly be grouped into the behavior of a single higher-level computational unit. But real neural networks can exhibit much more complicated forms of abstraction, because individual neurons can contribute simultaneously to many different functions.

A central question, then, is when two descriptions at different levels of abstraction can refer to the same underlying process. Said Icard, “If you have two causal models that are formulated in different languages with different variables, when do you want to say that one is a more abstract description of the same underlying causal reality than the other? This is called causal abstraction and we build on existing theory, adapting it to the study of neural networks in particular.”

In a study published in 2021, Icard and his collaborators—led by then-Ph.D. student Atticus Geiger, now a leading researcher in the field—showed that a BERT-based language model internally implements elements of a logical reasoning system. For example, the model learned an internal algorithm for performing complex logical inferences involving quantifiers (“every,” “some,” “not,” etc.) and negation.

An example that recently generated attention comes from the paper “Arithmetic in the Wild” by researchers at San Francisco-based Goodfire AI, led again by Geiger. The researchers studied how a Llama-based language model reasons over cyclic concepts, for example: “What month is six months after August?” One might expect the model to calculate directly within a twelve-month cycle. Instead, the model first uses standard decimal addition: August is treated internally as month 8, after which the model calculates 6 + 8 = 14. Only then does it translate the result 14 back to “February,” as 14 = 12 + 2, with February as the second month.

The Llama-based model uses this method not only to reason about months, but also about weekdays and clock time. The model thus applies the same general calculation strategy to different types of problems, without explicitly being instructed to do so. Said Icard, “The paper is a beautiful example of using causal abstraction methods to uncover how a large language model reasons about cyclic concepts.”

Academic researchers like Icard can only do their experiments on open-source large language models like Llama by Meta and OLMo by the Allen Institute for AI. “The largest models we have worked with contain around ten billion parameters,” Icard said. “In our experiments, we go inside and change the weights or the activations in the neural network. We do not have access to the commercial models, but I do know that some companies, such as Anthropic and Google DeepMind, do have teams working on mechanistic interpretability of large language models. We regularly talk with both these groups.”

Researchers are now exploring whether mechanistic interpretability can help make large language models safer, more reliable and efficient, while also reducing unwanted biases. “I think there is promise for all of those,” said Icard. Major challenges remain, however, such as scaling these techniques to larger models and automating parts of the interpretability process that currently still depend heavily on human insight.

“Mechanistic interpretability will probably never reduce large language models to a few simple equations,” Icard concluded, “but it may gradually turn deep neural networks into systems whose hidden algorithms can at least partly be understood.”

Bennie Mols is a science and technology writer based in Amsterdam, the Netherlands.