TL;DR -- Interpretability might be way more tractable than we thought, and the guy who helped convince everyone otherwise is now saying we can understand models by directly studying neurons and their interactions after all. This essay rewrites fundamental assumptions about how neural networks learn and connect, with some of the best interactive visualizations I've ever seen. Studies Qwen3 32B, Qwen3.6-35B-A3B, and Kimi K3 -- along with convolutional vision models. For years the standard view in AI interpretability has been that neural networks are a tangled mess. Individual neurons seemed to respond to random unrelated stuff (famous example: one neuron fired for both cat faces and car fronts), and the explanation was “superposition,” the idea that networks cram unrelated concepts into the same neurons to save space. If that’s true, you can’t just look inside a model and read it. You have to build a whole second model to untangle it first, which is a big part of why the field went all in on sparse autoencoders. Nick Cammarata (ex-OpenAI), one of the people who originally helped make that case, now says it was mostly wrong. The cat/car neuron? It’s a V-shape detector: pointy cat faces, pointy car grilles. The “mess” in early models came from weak architectures and buggy visualization tools, not from networks fundamentally packing junk together. Fix those and neurons get clean. When neurons do cover a few things, it’s usually because they branched out to related stuff over training, not random mixtures. His team’s early estimate is that roughly 90% of neurons in a modern Qwen3 32B language model have one main job, with similar results in much bigger models up to Kimi K3's trillion-parameter scale. He even suggests this feature separation becomes more defined as scale increases. Why this matters: if it holds up, we may be able to understand AI models by just studying them directly, the way you’d study any natural system, rather than first reverse-engineering a decoder for them. Bigger models actually seem easier to read, because their neurons are more specialized. And since AI agents can now do a lot of this investigative grunt work at scale, the author thinks frontier models could become broadly understandable in months rather than decades. For safety, that’s a big deal. Most arguments for why alignment is hard assume we can’t see inside the models. If we can, we can spot problems (say, deceptive reasoning or a bad training environment) and make targeted fixes instead of hoping we got everything right in advance. I know this is a long essay, but I think their upcoming technical paper will become a seminal work in the fields of interpretability and learning in neural networks (particularly in vision models). Even if you don't read it, the interactive visualizations and network architecture breakdowns are definitlely worth 5-10 min. Especially the ones towards the very bottom of the page. Let's discuss!   submitted by   /u/returnity [link]   [comments]