- 4 to -2.5
- change in distinct behaviours, pilot vs ten-run median
- p = 0.0097
- rank test, ten campaigns per arm
- 125 to 141
- covered cells over a six-hour campaign
Since November 2025 I have been building an LLM-driven tester for 5G protocols. The model reads the specs, proposes odd message sequences, and a real testbed runs them. I’ll write about the whole thing properly later. This post is about one small piece, because it taught me something I keep seeing elsewhere.
The piece is memory. A fuzzer that forgets what it already tried will keep trying the same things. So I gave the generator a memory: a short list of the test cells it had already covered, and an instruction to avoid them.
- 1Generate test sequences
- 2Run them on the testbed
- 3Score each result
- 4Record which cells were covered
- 5Put a hint about that in the next prompt
The pilot looked great
I ran one campaign with memory and one without. Duplicate tests dropped from 50% to 33%. The number of distinct behaviours I found went from 12 to 16. I was ready to write the paragraph.
Then I ran it properly: ten campaigns per arm, a rank test, an effect size. The result flipped. With memory, the median number of distinct behaviours fell from 15 to 12.5 (p = 0.0097). The first result was noise, and I had nearly built on it.
Why it hurt
I spent a while reading about this and the answer is not exotic. If you write “do not do X, Y, Z” in a prompt, you have put X, Y and Z in the prompt. Models drift toward what is in front of them. My list also grew every round, so the context got longer and the model repeated itself more.
The fix was to stop describing the past and describe the frontier instead: a short “here are the cells you have not tried yet” list, placed at the end of the prompt. It carried the same information as the old list.
Already covered: cell 3, cell 7, cell 12, cell 14, cell 19 … Do not repeat these.
Untried: 51 of 52 cells. Pick your next sequence from these.
What I can claim
On the improved setup, memory was statistically neutral on every metric I tracked. The no-memory baseline was already producing almost no duplicates, so there was little left to remove.
Where it did help was over a long run. In a six-hour campaign that carried the frontier forward, covered cells went from 125 to 141, and the tracker grew from 633 to 1,070 tests. So memory pays off across a campaign, not within a single run, and I’d be wrong to sell it as more than that.
What I took from it
- One run per arm told me the opposite of what ten did. Run ten.
- Tell the model where to go.
- Write down the neutral result too. It stopped me from building the next experiment on a result that wasn’t there.