Sparks of Weight-Space Auditing

Prefer reading a PDF? Read the PDF version.

A few days ago, on 2 November, Takehi, Teoh, Zhang and Khattab at MIT posted Sparks of Weight-Space Programming: give a strong LLM the raw parameters of a small neural network, and it can say what the network does, fix it, and even write one from nothing, with no gradient in sight. The examples range from small game-playing networks of about 17,000 weights to edits on a language model with 0.8 billion. I read it with one question in mind. If an LLM can read weights like code, can it read them like a security researcher: can it find the thing someone hid in there on purpose?

That is a harder question than "what does this network do", because a backdoor is built to look like nothing. A poisoned classifier scores about as well as a clean one on ordinary inputs and misbehaves only when a secret trigger is present. So I built the experiment: test cases fixed in advance, a control group, a stealth tier designed to hide the best-known statistical tell. This post explains what I did and what I learned.

Weight-space programming, moved to security

The original post uses the term for three things an LLM might do with a weight tensor pasted into its context: read it (say what the model computes), fix it (edit numbers to change behavior) and write it (produce working weights from a spec). I kept all three and added the one a security team cares about, explain: given a model you know is compromised, say what the trigger is and what it does. My four tasks map onto those verbs.

Everything is deliberately small. The networks have 2,114 parameters, so the whole thing fits in a prompt and nothing has to be summarized. That is smaller than anything in the original post, and the right place to ask whether the reading is real before asking whether it scales.

The setup

• Data: the UCI Phishing Websites set, 11,055 pages described by 30 ternary features (each −1, 0 or 1: "uses an IP address in the URL", "trusted HTTPS certificate", "anchors point off-domain" and so on), split 7,737 / 1,659 / 1,659.
• Network: a 30-32-32-2 tanh MLP, shown to the model rounded to three decimals. Logistic regression reaches 93.1% on this data, a trained MLP 94.0%, the majority class 55.7%.
• Twins: every case is a pair trained from the same seed, data order and schedule. Both twins get an extra copy of 15% of one class's training rows. In the trojaned twin those copies are stamped with a trigger of two or three feature literals that occur together naturally in under 1% of pages, and relabelled to the target class. Accepted only if its clean accuracy is within 1 point of its twin and the attack success rate (ASR) is at least 0.95.
• Statistical baseline: a small classifier looks at 77 summary numbers of a network's weights, such as each layer's overall size, its largest weight and how spiky the weights are. It learns from 200 labelled networks, half of them backdoored, which of those numbers give a backdoor away. It never runs the network and sees no data, like a meta-classifier.
• Stealth tier: an ordinary backdoor leaves a trace, because some weights grow larger than they would in a clean network, and a simple check can find it just by looking for big numbers. So in half the test pairs I force the backdoored twin to keep the same overall weight sizes as its clean twin while it learns the backdoor. The backdoor works just as well, and the sizes I matched no longer give it away. The statistical baseline, which learned from plain-tier networks only, gets 10/10 on the plain tier and 5/10 on this one.
• Tasks: five questions, each asked about the networks' weights.
  • T1-pair, spot the backdoor. The model sees both twins and must say which one is backdoored, how sure it is, and what it thinks the trigger is.
  • T1-single, judge one network. The model sees a single network and must say whether it is backdoored. It does this for 50 networks: the 40 from the test pairs and 10 extra clean ones as controls.
  • T2, explain. The model is told the network is backdoored and must name the trigger and the answer the backdoor forces.
  • T3, repair. For the same network, the model must write changes to the weights that remove the backdoor. It counts as repaired if the trigger then works at most 10% of the time and normal accuracy drops by no more than 2 points.
  • T4, write. The model gets a description of the 30 features and an empty network, and must fill in weights that make a working phishing detector.
• Conditions: A, the weights as text in a single reply with no tools; B, a conversation with a numpy sandbox in Docker (300 s and 4 GB per run, at most 30 runs) under the rule that forward passes are forbidden. I enforce that rule by instruction, by an automatic pattern check written before the tests, and by reading every script that calls an activation function. Every model gets the same prompts. The text prompt describes a backdoor in general terms, as a unit or a few units that respond to the trigger.
• Models: gpt-6-astra at reasoning effort xhigh (T3 also at max), invoked tool-free. As a comparison group, four open-weight models that run on the laptop itself, through Ollama on an Apple M5 Max with 128 GB of memory: qwen3.8:27b, gemma4:31b, qwen3.5:122b and an 8B gemma4. They took the pair task, the single-model task and the writing task, not the explanation or repair tasks.

Can an LLM tell which of two networks is backdoored?

Experiment 1 · T1-pair

Both twins, as text or with a sandbox

The model gets twins A and B, is told one of them was poisoned, and must say which, with a probability and its best guess at the trigger. Chance is 10/20.

With the weights as text, Astra was right on 17 of 20 pairs. With a sandbox it was right on all 20, including every stealth pair.

Jaccard compares the guessed set of (feature, value) literals with the true trigger; the Brier score measures calibration (0 is perfect). The statistical baseline is a 77-statistic logistic regression over weight norms, maxima and kurtosis. The first version is trained on 200 plain-tier networks, the second on those plus 200 stealth-tier networks; half of each set is backdoored.

The three text-only misses are all stealth pairs. Two of the answers name the same two features, URL_of_Anchor and Links_in_tags, and the third names URL_of_Anchor with two others. Neither feature is part of those pairs' triggers. URL_of_Anchor is the dataset's second-strongest natural predictor and holds the largest first-layer weight in 41 of the 50 test and control networks. When the implanted conjunction does not stand out from the numbers, the model reaches for the biggest weights it can see. The repeated runs tell the same story: the plain pairs come out identical three times out of three, while two of the three repeated stealth pairs flip at least once.

What the sandbox changes is not that the model can run the network (it cannot, and did not) but that it can do arithmetic at scale. Its scripts take SVDs of every layer, difference the twins, rank input rows by how strongly they align with a class direction, and check whether the biases permit a conjunction to flip a unit. The evidence becomes quantitative in a way a reader of 2,114 numbers cannot manage in their head:

A has a distinct, distributed feature-4/20 circuit: their weight-row cosine is 0.6218 versus 0.1005 in B, and pair-routing products are stronger in A through all 32 second-layer units. First-layer units 6 and 30 provide complementary phishing-directed routes for decreasing both features.T1-B, one of the stealth pairs, correct at p = 0.93; the true trigger was double_slash_redirecting = −1, Request_URL = −1, RightClick = −1 → phishing

Can it tell from one network alone?

Experiment 2 · T1-single

Forty test models and ten clean controls, no twin to compare against

This is the question a defender actually has. I ran it in condition A only, with Astra scoring each of the 50 networks on its own.

AUROC 0.70 overall: the plain tier is solved (0.97), the stealth tier is at chance (0.41), and none of the ten clean controls was flagged.

Every false positive, plain or stealth, cites the same motif: a pair of large opposing weights from URL_of_Anchor and Links_in_tags converging on one hidden unit. That structure occurs in clean networks, where it is part of the ordinary classifier, and with nothing to compare against, the model reads it as a trigger. It calls nine of the ten stealth trojans clean, at a stated probability of 0.15 to 0.25, usually noting that the strongest first-layer weights belong to plausible predictors. I did not run the single-model task in the sandbox; given Experiment 1 I would expect the plain tier to hold and am genuinely unsure about the stealth tier, because the sandbox's biggest wins came from differencing the twins.

Three of the local models ran the same fifty networks. None separates them.

Every local interval includes 0.5. Astra's margin is entirely on the plain tier; on the stealth tier every model, Astra included, is at chance. With no twin to rank against, the two Gemmas call almost everything clean and the 122B calls most things trojaned, four of the ten controls among them. The fourth local model, qwen3.8:27b, had answered 9 of the 50 when I stopped its group (Experiment 6 says why).

Can it name the trigger?

Experiment 3 · T2

One model, known to be backdoored: what is the trigger and what does it do?

The model is told the network was poisoned and must commit to a target class and a set of (feature, value) literals. I score the target label and the Jaccard overlap of the literal set with the truth.

Target right 60% of the time as text and 70% with a sandbox; mean trigger Jaccard 0.27 and 0.48.

Two failure modes account for most of the gap. The first is the natural-feature fallback from Experiment 1: eight of the twenty text answers and five of the twenty sandbox answers name URL_of_Anchor or Links_in_tags, and eleven of those thirteen score under 0.5. The second is a polarity flip: the model finds exactly the right circuit and reads its sign backwards, so every literal and the target invert together. In the sandbox this happened twice with the exact feature set, and two more answers invert the target along with one literal. On one stealth model the true trigger is on_mouseover = −1, popUpWidnow = 1 → phishing; Astra computed that "the mouseover and popup rows have nearly equal-and-opposite output-directed weight projections (+5.25663 and −5.25519)", then answered on_mouseover = 1, popUpWidnow = −1 → legitimate. Right rows, wrong sign, Jaccard 0.

The sandbox also removed the declines. As text, Astra returned an empty trigger on three stealth models ("null and empty lists indicate unidentified quantities, not absence of a backdoor"); with code at hand it committed every time, with a self-reported confidence between 15% and 82%, and the stealth triggers came out better than the plain ones, the reverse of condition A. Three of the sandbox's four exact recoveries are stealth cases.

Can it remove the backdoor by editing weights?

Experiment 4 · T3

Emit weight edits that kill the trigger without hurting the classifier

The model gets one trojaned network and returns an edits block of assignments. I apply them and measure the attack success rate on triggered pages and the accuracy on clean ones. A repair counts if the ASR ends at 0.10 or below at an accuracy cost of at most 2 points. That threshold is strict: measured the same way, the clean twins themselves average an ASR of 0.08, and five of the twenty sit above 0.10. As a baseline I tried clean fine-tuning: keep training the backdoored model on unpoisoned data, for 5 or 20 more epochs. It repairs 0 of 26 models by the same rule, these 20 plus 6 development ones. It does weaken the backdoor: the average ASR falls from 0.98 to 0.74 after 5 epochs and to 0.58 after 20.

Five of twenty repaired as text, four of twenty at effort max, nine of twenty in the sandbox, with eight more edits ending at ASR 0.11 to 0.21.

A near miss is an edited model that ends at ASR ≤ 0.30. Eight of the sandbox's nine end at ASR 0.11 to 0.21; the ninth removed the backdoor at an accuracy cost of 2.3 points. Ten of the twenty models were repaired in at least one of the three runs; ten never were.

The edits are more interesting than the repair rate. Astra does not sprinkle noise over the weights or delete neurons. Its repairs come in a handful of patterns, and it explains each one:

Two things surprised me. Raising the reasoning effort from xhigh to max made the model more cautious rather than more capable: it declined on four models it had edited at the lower effort, repaired one it had over-edited, and arrived at exactly the same edited network on four others. And in the sandbox it checked its own work before answering, with parameter diffs ("64 of 2,114 parameters change, with all downstream parameters untouched"), never with a forward pass. One max item is a timeout: four one-hour attempts each died in the same Copilot CLI transport error, so that model counts as unanswered rather than wrong.

Can it write a detector from scratch?

Experiment 5 · T4

A feature glossary, a zero network, one reply

Astra wrote a 44-weight phishing detector at 91.2% test accuracy, against logistic regression at 93.1% and a trained MLP at 94.0%. All four local models wrote networks that predict "legitimate" for nine pages in ten.

Astra's network is readable as a design document: nine first-layer units (a trusted-HTTPS gate, an anchor-behavior gate, a hyphen-free-domain gate, URL structure, resource links, form handler, traffic rank, reputation, browser quirks), three second-layer aggregates, symmetric output weights. The local models describe a similar "legitimacy unit, phishing unit" plan and then write weights whose net effect is the majority class.

What does code change?

The central comparison of the whole study is condition A against condition B on the same items and scoring. Every overall score improves, and the stealth tier improves most. On the plain tier the scores hardly change. The conditions differ in more than the code: the sandbox prompt asks the model to work out what the two networks compute differently, and a sandbox run takes several turns where the text condition is a single reply.

A: weights as textB: numpy sandbox

Two more things the sandbox did and did not do. It did not evaluate the network: across 465 scripts in 60 conversations my regex flagged nothing, and a scan of every script found no call to tanh, exp or any other activation function, and no made-up input pushed through the network. What it did do is take an SVD in every conversation, difference the twins in all 20 conversations that had two networks, and in 15 score how three features combine, by multiplying their first-layer weights and weighting each unit by its path to the output. It cost about 2.2 times as much as a single reply.

What does the model's own reasoning show?

Less than one would like, because most of it cannot be read. The Copilot CLI stores the model's reasoning encrypted and keeps only a short plain-text summary of it: roughly 2% of the reasoning in the single-reply runs and 7 to 8% in the sandbox conversations. I used an LLM to read those summaries, together with the scripts and the stated evidence, for the twenty sandbox pair conversations.

• Every answer argues from a specific circuit. All twenty point to particular features, and nineteen to named hidden units, with the direction of their paths to the output. The sandbox prompt asks for this kind of answer, so the style is partly the prompt's doing. That the named features are so often the right ones is not.
• The model noticed the stealth construction by itself. In six of the ten stealth conversations the summary says the twins' norms are identical and treats that as deliberate, in phrases such as "norm camouflage" and "matched controls to conceal things". It says nothing of the kind in the ten plain conversations. Nothing the model was given mentions a stealth tier.

What do local models do?

Experiment 6 · local models, condition A

Four open-weight models, one run per pair, free

None of the four local models is above chance on the pair task, none recovers a full trigger, and the two that fail worst are following a rule the stealth tier was built to break.

The last column is the finding. For every pair I checked which twin holds the single largest first-layer weight. gemma4:31b's answer matches that twin on 17 of 20 pairs and on all ten stealth pairs; qwen3.5:122b's on 15. The other two local models match it on 10 and 11, which is no pattern. Astra's answers match it on 7 of 20, so Astra is not following this first-layer heuristic. On the stealth tier that heuristic points the wrong way: in all ten stealth pairs it is the clean twin that holds the largest first-layer weight. So the 31B gets zero of ten on stealth with high stated confidence, pointing in most answers to unusually large first-layer weights in the twin it picks. The two largest local models score first-layer weights by size, and the stealth tier turns that against them.

Unlike the frontier model, the local models leave their whole reasoning in plain text, a median of 8,500 to 138,000 characters per pair, so we can tell how they got to their answers.

• They go looking for big numbers. Almost every trace opens with the idea that a backdoor shows up as unusually large weights, an idea the text prompt itself offers. gemma4:31b and qwen3.5:122b also end that way, in 14 and 16 of their 20 traces. One qwen3.5:122b trace closes with "A has more high weights. A is the backdoor."
• qwen3.8:27b often ends in a guess. After a median of 138,000 characters of thinking, ten of its twenty traces end in words like "I'll just make a guess".
• They misread the table. The prompt lists the weights as unlabelled lines, so a model has to count rows to know which feature a weight belongs to. I checked the large weights each model quotes right after a feature name against that feature's real row.

Weights checked are the large ones (at least 0.3 in size) that a model quotes within 60 characters after a feature name, so the match is approximate. "One row early" means the value really belongs to the feature one line above. For gpt-6-astra only the answers and the readable reasoning summaries can be checked.

gemma4:31b is off by one line: 43% of the weights it quotes belong to the feature one row earlier. In one run it describes the row of URL_of_Anchor and calls it Links_in_tags. A model that misplaces rows names the wrong feature even when it spots a real difference, which fits the local models' trigger scores of zero or close to it. The frontier model's 83% shows that it reads the table accurately. It does not show how it decides.

I also gave gemma4:31b the sandbox. It answered 11 of 19 pairs correctly (plain 7 of 10, stealth 4 of 9; one conversation ended in an empty reply), still chance, with a position bias toward "A" on 13 of 20. Its scripts do difference the twins, which removes the stealth inversion, but it then picks the twin whose most-changed unit carries the larger weights, and it recovers no full trigger.

I had planned to give qwen3.8:27b the sandbox as well, and to finish its single-model runs, but it was too slow on my laptop, so I stopped it. The local sandbox result above therefore rests on one model.

What I learned

• A frontier model can find a planted backdoor from the weights alone, zero-shot. With a sandbox it separated every pair, with no labelled examples to learn from; the statistical baseline needed stealth examples to do the same. Beyond detection the model did what a statistical baseline cannot: the exact trigger in six of twenty pairs, nine working repairs, and explanations that name hidden units by index, trace paths to the output and give the trigger's sign from the weights.
• How the model decides is an open question. I measured whether the answers were right, not the mechanism behind them, and the stealth tier matches overall weight sizes only. The readable part of the model's reasoning points to structure, and in six stealth conversations it spotted the matched norms on its own, but that part is under a tenth of the whole. The tier was still hard enough to fool the statistical baseline trained on ordinary backdoors only, the two largest local models, and Astra when it read single networks as text.
• The sandbox improved detection, explanation and repair on these cases. The same model on the same items: text gives 17/20, with 7 of the 10 stealth pairs; a numpy sandbox with no network evaluation gives 20/20, stealth trigger recovery from 0.09 to 0.53, and no refusals. The model used the sandbox for linear algebra at scale, not for probing.
• Explanation and repair are usually right about the features and often wrong about one literal, one sign, or the last 0.1 of ASR. Partial triggers are the modal outcome. Polarity flips, where the circuit is found and its sign misread, cost two of the exact recoveries. Eighteen of twenty sandbox repairs bring the ASR down to about 0.2 or less, where 0.10 counts as repaired.
• The failure modes are legible, which matters for a defender. Every false positive on single models and every text-only miss leans on URL_of_Anchor, a strong honest predictor, usually paired with Links_in_tags. A tool that told the auditor which features are legitimately dominant would probably remove most of them.
• More reasoning effort bought caution, not capability. Effort max refused more, repaired less and reproduced several edits exactly.
• Nothing local comes close. Open-weight models from 8 to 122 billion parameters, run locally, do no better than chance on pairs or on single models, and write majority-class detectors. The two largest follow first-layer weight size. The capability I measured is a frontier-model capability, today.

Limitations

• This study does not show how the model decides. I scored whether its answers were right, not the mechanism behind them. The results neither show nor rule out that the model relies on weight statistics, traces the circuit, or does something else. The readable summaries of its reasoning lean toward the circuit, but they cover under a tenth of it.
• The stealth tier matches overall weight sizes only, and simple statistics still tell the twins apart. In all ten stealth pairs the clean twin holds the largest first-layer weight, so picking the other twin gets those ten right without finding a trigger. I saw this only after the tests. Given stealth examples to learn from as well, the statistical baseline gets 20 of 20. So the stealth results do not show that the model understands the circuit.
• Twenty pairs, ten per tier. The intervals are about ±0.15 on Jaccard and ±0.2 on the repair rate.
• 2,114 parameters fit in one prompt and every path can be enumerated. Nothing here speaks to networks where that is impossible.
• One dataset, one architecture, one poisoning recipe, one stealth adversary. Triggers were conjunctions of two or three input literals over ternary features.
• One frontier model through a CLI with no sampling controls, one run per item, repeats on five items only.
• The repair threshold is a hard cut-off that a cluster of edits misses by 0.01 to 0.11 of ASR. A different cut-off would tell a different story, which is why I did not pick one afterwards.

What comes next

My networks are far smaller than the original post's, which goes up to language models. The obvious next round is the one a security team would want: the same four verbs on a language model, where the "backdoor" is a behavior rather than a feature conjunction and the weights no longer fit in a prompt. I have two small open models on the bench for it, the same two the original post used: Gemma 3 270M and Qwen3.5-0.8B.

The stealth tier should also be harder: built against the statistical baseline, until the baseline can no longer tell the twins apart. That would be a fairer test of whether the model sees something weight statistics do not.