Run it again: what 26 audits record about repeating a backtest

2026-09-04. Every figure quoted here is a field in a published audit JSON, and the listed audits are linked in place.

A backtest is one run, and whether a second run on the same data would produce the same trades is a different question. Every audit we publish carries a single field for it, called non deterministic. Of our 26 default audits, 3 record that the run does not repeat, 19 record that it does, and 4 carry no answer at all. The field is one word long and it changes the meaning of every other number on the page, because a result that does not repeat is one draw rather than a measurement.

Where the answer comes from

Two routes can set it, and they are not equal.

The tester journal. Our parser counts the lines in which the expert prints a random number of its own. A count above zero sets the field. This is one of the reasons our protocol makes the journal a mandatory upload rather than an optional extra, alongside exact pairing of entries to exits.
A standing note in our catalog. We can set the field for an expert by hand. That route only ever raises the answer and never lowers one, so a note can add a warning and no note can remove one.

Three of the twenty six do not repeat

Prop Firm Gold EA is the loudest case. Its journal carries 6,030 randomization prints across 9,538 trades, so the count alone decides the field. Range Breakout EA carries 2,481 prints across 1,525 trades, and it is also the one case where we ran the experiment instead of reading the journal. Two runs with a byte identical settings file produced 175 deal rows each, of which 174 differ. Type, direction, volume and order were identical in all 175, so what moved was the price, the time and the money rather than the plan.

Gold Atlas is the third, and its journal prints 0 randomization lines. The answer there comes from the standing note, not from the count.

A count without a field, and a field without a count

Those two ends of the catalog are worth putting side by side, because each shows one half of the machinery failing to meet the other.

Market Anomalies EA, which we keep unlisted because it trades currencies rather than gold, has 229 randomization prints in its journal. The flag list on its page says so, in an entry titled Trade randomizer detected. The non deterministic field is missing from that audit entirely, so the standing warning we print on a non deterministic audit never appears there. The reason is mechanical rather than editorial. Four of our 26 audits were built by engine version 0.2.0, before the field existed, and those four are Market Anomalies EA, Pulse Engine, Quantum Emperor and Waka Waka.

Gold Atlas is the mirror image. It carries the field and no flag, because the flag needs a print count above zero and its journal has none. A reader who trusts only the flag list would miss the warning on one page, and a reader who trusts only the field would miss it on the other.

What a missing journal would do

The field is built from the print count, and a run without a journal has no count. An absent count is not a zero, but the conversion into a yes or a no reads it as a no. The silent failure runs in the wrong direction, because an audit with no journal is published as repeatable rather than as unknown.

That is not theoretical. Range Breakout carried no journal in our catalog when its behaviour was first measured, and the entry read deterministic while two runs of it were visibly disagreeing. The standing note on that entry exists for exactly this reason. Today the journal is in place and prints 2,481 times, so both routes agree and the note is now redundant rather than wrong.

All 26 default audits record that a journal was provided when they were built, so no published page gets its answer from an absent count. The audit page is also stricter than the field here, because it prints the randomizer count only when a journal exists and shows n/a otherwise.

Two journals held two runs

Logan and Waka Waka are the two audits whose journal contained 2 test runs rather than one. Both pages say so and both state that only the last run was used. The other 24 carry exactly 1. It is a small field and it answers something a reader would otherwise have to take on trust, which is which of two runs the numbers came from.

What a zero does not prove

23 of the 26 print zero randomization lines, and that is not evidence that a second run would match. It is the absence of one specific signal in one journal. An expert can draw on something it never prints, and a window can stay identical by chance while a longer one diverges. A test of this kind can find randomness. It cannot establish its absence.

So read the field the way it is built. A yes means we found a randomizer. A no means we did not find one in this run.

Read it before the profit line. On a non deterministic audit the drawdown, the win rate and the net all belong to one draw, and a second run would print different ones.
Check the flag list and the field. They come from different routes, and our own catalog holds one page where each is present without the other.
Ask how many runs the journal held. Two of ours held two.

Every count above is public in the audit files linked in place. Holding your own tester report? The browser check is free.

The honest limits. The counts above cover 26 default audits but only 25 distinct runs, because Quantum Queen and Quantum Queen X are the same measured run published under two slugs and both record the same answer. The print count depends on what an expert chooses to write into the journal, so a randomizer that stays quiet leaves no trace for us to count. And the one repeat experiment we ran covers a single expert over a single window, which is enough to show that one robot draws and not enough to say anything about the rest of the catalog.