These four came out of the same project, an LLM-driven tester for network protocols. Each one started as a number, a comment or an assumption I had not checked.
The eight crashes that were zero
At some point in my notes I wrote down that a training dataset contained eight crashes. It was the headline of the summary, and I had flagged it as the strongest result so far. It was wrong.
When I went back and queried every results database, there were 42,752 result rows and no crashes. The “eight” came from reading a database file that was empty, 0 bytes, next to the one that had the data. The scoring code can only emit a crash verdict when a crash status or a signal shows up in the logs, and the logs showed neither.
I fixed the note, and then I did something I should have done earlier: I stopped trusting any positive I had not seen twice.
Most positives were flakes
I re-ran every distinct sequence, 2,236 of them. Of 171 sequences that had been labelled interesting, only 34 reproduced. About 80% of my positives were not reproducible.
It got worse with scale . The positive rate was 18.0% on the first 355 queries, then 4.8%, 5.1%, and finally 0.3% on a run of 12,345. Adding roughly thirteen thousand queries gave me 67 new training examples.
Those positives were going to be the reward for training, and a noisy reward makes a model learn the wrong thing. So the harness now replays each sequence several times and scores it with a conservative lower confidence bound instead of a single verdict.
- 1Sequence scores a positive
- 2Replay it several times
- 3Count how often it reproduces
- 4Take the Wilson lower bound
- 5Use that as the reward
| Replays | 100% | 90% | 80% | 50% |
|---|---|---|---|---|
| 5 | 0.57 | 0.38 | 0.38 | 0.12 |
| 10 | 0.72 | 0.6 | 0.49 | 0.24 |
| 20 | 0.84 | 0.7 | 0.58 | 0.3 |
| 50 | 0.93 | 0.79 | 0.67 | 0.37 |
If every sample in a training group gets the same reward, the gradient is zero. My first smoke test with a tiny model hit exactly this, which is why the real run starts from an already tuned adapter.
More spec made it worse
The obvious way to get an LLM to write good protocol tests is to show it the relevant paragraphs of the specification. So I tested it: the same generator with and without spec text in the prompt, two seeds each.
Without the spec, 32 of the tests were valid under my schema. With it, 2. On the second seed it was 29 against 0. That is a large effect (Fisher p around 1e-60).
The reason was boring. My schema wants machine-style identifiers. The specification writes names the way humans do, in capital letters with spaces. Shown a page of that, the model copied the format it saw, and every test failed validation.
REGISTRATION REQUEST
registration_request
When you need structured output, the model takes the format of your context as an instruction too. Correct context in the wrong format can do more harm than no context at all. The same pattern showed up elsewhere: when I compared a hand-built playbook of attack ideas with a generic prompt, I could not detect any difference (p = 0.49), and the direction flipped between seeds.
The sixty fake messages
A comment in my extraction code said that including one particular section of the spec makes the model invent about sixty non-existent messages. Everyone, including me, treated this as a known fact, and the section had been excluded for weeks.
Nobody had measured it, so I did. That section produced 59 candidate messages. Every one of them reused an identifier that was already in my 61-message baseline. None were new and none were made up.
The failure was redundancy: the model re-derived things it already had. My deduplication step then hid it, which is why a naive count came out as 61 against 61 and looked like “no effect”.
What I do now
- Replay before I reward, and check the thing measuring the model before I blame the model.
- If a code comment states a number, find the experiment behind it or delete the comment.
- Look at the data before the dedup step as well as after.
- Write the correction where I wrote the claim, and write it the same day. The embarrassing version of this story is the one where the number goes into a write-up.