I made a couple of posts about weird neurons inside Qwen that I'd found and lots of people were asking how it was done so I thought it would be useful for people to have a guide. Generally this was inspired by justinrhowell's neuron-steering repo, although I didn't use his code directly I took a couple of the approaches so props to him for the work! What is neuron steering? Each layer of Qwen has an MLP block with thousands of neurons. Qwen3.5-4B has 32 layers with 9,216 neurons each, so roughly 295,000 in total. Neuron steering is basically choosing one neuron and adding a fixed amount to it (the 'dose') when asking for an output from the model. This is not finetuning, or fancy prompting - it's basically just a really simple form of activation steering that involves one neuron only. But interestingly, some neurons change the model’s behaviour a lot! To keep track of what neurons you're editing, they have addresses. “L11 N8147” means the 12th layer, neuron 8147 - this is the naming convention justin used, and is basically the same as the neuronpedia address (QWEN3.5-4B · MLP Neurons - Layer 11 · 8147 ｜ Neuronpedia). How do I actually do it? The easiest way will honestly be to just ask your agent of choice to set up a neuron steering harness for you (it's not that complicated), but if you want to do it manually: Load the model in PyTorch with transformers. Find the model’s list of decoder layers, then add a hook on the down_proj of the layer you want. The hook will add a fixed value to that one neuron at generation time for every token:amount = dose * median_residual_norm[L] / W_down[:, n].norm()def pre(mod, args): x = args[0].clone() x[..., n] += amount return (x,) + args[1:]layers[L].mlp.down_proj.register_forward_pre_hook(pre) Generate text and see what it comes out with (often helpful to have the base model responses as a baseline) The value for 'dose' is basically how much you want to push the neuron/modify its activation. 1 is a big push, 0.75 usually works pretty well. Going above 1-1.5 usually gets into gibberish territory. median_residual_norm[L] is how big the model’s hidden state normally is at layer L - you can run a few paragraphs through the model once, measure it at each layer, and take the median. Scaling by it means dose 1 is roughly the same strength of push at any layer. To properly chat with a model you've tweaked in llama.cpp, you can save the change as a control vector and load it with --control-vector. How to find interesting neurons Most neurons don't actually seem to do anything especially interesting when you modify them (as far as I can tell), but you have a few options to find the interesting ones: 1. Random search (easiest option). literally just pick random neurons from the middle layers, push each one at dose ±1 (both polarities are useful, sometimes interesting neurons only do something at one polarity), and ask a few simple questions (e.g. "who are you?", "write me a haiku"). Then you basically have to go through and check all the responses to see if any of them are interesting or not - strongly advise having an LLM do this checking for you otherwise you'll get extremely bored. This method is slow and mostly just gives you junk, but it’s also how I found some of the weirder neurons so far like one that makes Qwen think it's a TV channel. 2. Gradient shortlist + LLM judge (still exploring this process). This is an approach I took from justin's repo, but I'm still exploring it and some of my results have been inconsistent in some interesting ways, but the gist is: Use a system prompt to make the model write answers in a specific style or set of styles (e.g. maybe you are hunting for a slang neuron, so you ask it to write in slang) Then with no system prompt, you compute for every neuron how much a push would make those answers more likely than normal answers. (I could share a script for this but honestly if you just paste this paragraph into your LLM it will be able to build this) Keep the top 40 neurons for each style and push each one at dose +/- 1 or 0.75 (polarity suggested by the gradient) to see what it does. If you want to automate the output reviewing, make sure you don't tell the judge-LLM what style difference it's supposed to be looking for. 3. Activation mining (have only personally played around with this a little). Basically the gist here is that you make the model write a few hundred short responses on lots of different topics, and record every neuron’s activation. You then try to find neurons that only fire on one topic, have an LLM label them from their top examples, then push them. This approach seems to be quite good at finding “topic detectors” (e.g. dinosaurs, astronomy, construction) but I'm not sure if it's possible to get at more meta-concepts or interesting behaviours through this approach? I think I find the topic detectors less interesting because the model just starts shoehorning dinosaurs into the conversation, rather than displaying any particularly interesting behaviour. For all of the methods above you if you think you've found an interesting neuron you should also try to check other neurons in the layer to see if yours is truly special or not - randomly select 3-5 other neurons in the same layer and push them by the same amount, if they do something similar then it's likely not due to that neuron specifically. Also make sure to try different doses, sometimes tuning or flipping polarity leads to interesting results. And also trying out lots of different prompts to get a feel for exactly what it is the neuron is doing is important - you can also look up the neuron directly on neuronpedia. So, there we go that's pretty much everything. I'm not a proper research scientist so apologies for any ignorance in this post. Please let me know if you find anything interesting!   submitted by   /u/Tasty-Ad-3753 [link]   [comments]