Why do LLMs make stuff up? New research peers under the hood.
Claude's faulty "known entity" neurons sometime override its "don't answer" circuitry.
One of the most frustrating things about using a large language model is dealing with its tendency to confabulate information, hallucinating answers that are not supported by its training data. From a human perspective, it can be hard to understand why these models don't simply say "I don't know" instead of making up some plausible-sounding nonsense.
Now, new research from Anthropic is exposing at least some of the inner neural network "circuitry" that helps an LLM decide when to take a stab at a (perhaps hallucinated) response versus when to refuse an answer in the first place. While human understanding of this internal LLM "decision" process is still rough, this kind of research could lead to better overall solutions for the AI confabulation problem.
When a “known entity” isn't
In a groundbreaking paper last May, Anthropic used a system of sparse auto-encoders to help illuminate the groups of artificial neurons that are activated when the Claude LLM encounters internal concepts ranging from "Golden Gate Bridge" to "programming errors" (Anthropic calls these groupings "features," as we will in the remainder of this piece). Anthropic's newly published research this week expands on that previous work by tracing how these features can affect other neuron groups that represent computational decision "circuits" Claude follows in crafting its response.
