A Market With Nothing In It — Agent-Based Null Markets for Catching an Overfitted Optimizer

Every backtest on real prices has the same hole: the optimizer found a profit factor above two, and you cannot tell whether the market held the pattern or the search found the best of ten thousand accidents. Hammad Dilber's Agent Based Market Simulation (MQL5 24664, 16 Oct 2026) builds the control real data can never be — a market whose contents are known because nothing was planted in it — and points an optimizer at it. The library pairs with oos-lab and a walk-forward habit: oos-lab asks whether a winner survives unseen bars of the same series; this asks what the same search is worth on a series where the answer is known in advance. It is a tool for measuring your search, not a strategy, and the author is careful about the difference.

1 / What the object actually is

Two models. The centre is the minority game (Challet & Zhang 1997, from Arthur's El Farol problem): an odd number of agents each pick a side each step; the minority is paid. Each speculator holds a few fixed strategy tables mapping the last m history bits to ±1 and learns only which table to trust — it cannot invent a better rule. Producers (one fixed table, never learn) are why there is anything to exploit; trend followers are the injected bias, and at zero they plant nothing. Alongside it, the Gode–Sunder (1993) double auction: zero-intelligence traders quoting at random, scored against the maximum surplus a central planner could reach, computed exactly.

Two order parameters judge the game. Volatility per agent (variance of excess demand divided by N; 1.0 is the independent-coin limit) and predictability H/N (mean demand conditional on the history word, squared and averaged; zero says the history tells a newcomer nothing). Sweeping α = 2^m/N, the author's measured volatility minima are 0.5111 / 0.4313 / 0.4324 for m = 6/7/8 against the published 0.3374 — a gap the fine scan does not close (five populations, band 0.4113–0.5501, every point above published). The supportable claim, as the author states it, is that the minimum exists, depends on α rather than on m or N separately, and lands near but above 0.3374. The sharper test passes cleanly: at fixed α = 10, growing N from 27 to 1,639 holds volatility per agent between 0.849 and 0.878.

Making it look like a market takes one more ingredient. The plain game fails the Cont stylized-facts checklist (excess kurtosis +0.157, mean |return| autocorrelation 0.0013 on 40,000 steps, author). The grand-canonical variant — sit out while your best table scores below a benchmark ε, with ε subtracted from every score every step — gives excess kurtosis +0.7214, mean |return| acf 0.0752 (Ljung–Box p on |returns| underflows to zero), Hill tail index 3.97 against 6.17 for the independent-noise control, and participation averaging 97.6 of 1,000 speculators active. The benchmark is calibrated, and the band is narrow: across ε 0.0006 → 0.0024, excess kurtosis falls from +28.666 to +0.246 while participation moves by only about seven agents. The author states the consequence himself: the checklist was the fitting criterion, so passing it is not independent confirmation. The lag-one sign autocorrelations (−0.0019 / −0.0118) reject independence under Ljung–Box (p 0.00337 / 9.98e-17) yet price to almost nothing: fading one step scores t +0.02 / +1.20, earning at best 0.0060 of a standard deviation per step. Structure a test can find and a rule cannot use.

Zero intelligence prices the matching rule. Budget-constrained random traders capture 0.9597 of the planner's surplus (mean trade price 99.28 vs equilibrium 99.17, no value-destroying trades, price dispersion 29.14); remove the constraint and efficiency falls to −0.0228, with 29.9 of 60 trades per period pairing a buyer and seller who should never have met. Most of what looks like a market's intelligence is the arithmetic of its matching rule.

The payoff: run the search where the answer is known. The detector scans lookbacks in both directions on the first half of a generated series, takes the single winner, scores it on the second half, and keeps the best/worst out-of-sample scores the optimizer would throw away (the grid is antisymmetric: half of it scores positive on any series). Confirmed on 200 null markets: the winning rule averages in-sample t 1.672 and out-of-sample t 0.205 (SE 0.066, SD 0.932) — small, unit-scale, and 3.1 standard errors above zero, not zero; the author traces the residue to weak anti-persistence at lookback 20 (t −2.74). Detection (t ≥ 2) on the null is 3.5% (Wilson 95% 1.7–7.0%). Planted edges: at 20 followers (about 3% of the population in the earlier 20-market run) detection is 18.0% — genuinely there, missed four times in five; at 40 and 80 followers, out-of-sample t averages 31.5 / 47.5 and detection is 100%. The three smallest levels (0/5/10 followers: 3.5% / 1.5% / 2.5%) are indistinguishable from the null. That is a threshold between 10 and 20 followers, not the monotone staircase the 20-market run suggested.

Then the same thing through the Strategy Tester. The generated market becomes a tradable custom symbol at 1 step per bar (the bar-aggregation choice is itself measured — see §2), and a minimal momentum/fading EA is optimized over lookbacks 2–20 in both directions on the first 10,000 bars, zero spread. The optimizer's winner — lookback 18, fading, 958 trades — posts in-sample profit +$42.54, PF 1.12, Sharpe +6.79 (author's tester figures; only 3 of 20 parameter sets end the first half in profit). Carried to the half it never saw: −$53.86, PF 0.88. The winner made 4.4 points per trade, so a five-point spread (costing 47.90 over 958 trades at the author's conversion) erases the whole in-sample profit before the second half is reached. ABM_Test.mq5 prints 82 passed, 0 failed; the Python replica cross-check prints 83 passed, 0 failed, bit-for-bit on the integer path, with the numpy/scipy/statsmodels layer agreeing to a largest relative difference of 9.6e-13 — agreement that establishes the specification was implemented twice, the author notes, not that the model resembles a real market.

2 / What it does not claim — and where it fails

The author's own sentence carries the section: the null is a null only in three senses, and each is a choice you must hold fixed.

• Only up to the agents' memory. Rules reading a comparable window to the agents' 7 bits find nothing (momentum t −0.61 at lookback 5, −1.06 at 10). Longer rules find the generator's long-horizon anti-persistence: momentum t reaches −5.92 at lookback 800, variance ratio falls to 0.153 at 500 steps (plain game) against ≈1 for the control, and a wide scan's winner scores out-of-sample t +5.820, detected 10 of 10. In the tester, every one of 8 long fading sets profits in-sample and the winner keeps +$139.68, PF 2.59 out-of-sample. That is a property of the generator, recoverable by an honest procedure — which is the point — and harvesting it is not a trading edge.
• Only at the aggregation you build. At 30 steps per bar the plain market hands a fading rule out-of-sample t +22.66; at 1 step per bar the same market reads acf1 −0.0075 and out-of-sample t +0.31 against in-sample +1.38. "Which market to build is a measurement, not a preference." The shipped symbol default (30 steps/bar) is not the null used in the experiments (1 step/bar).
• Only for the rule class searched. A pass clears one failure mode — selection on an in-sample score — for the grid you actually ran. It says nothing about costs (the tester symbol charges zero spread), regime change, or real participants.
• Not perfectly null even inside its own fence. The 200-market out-of-sample mean of 0.205 is 3.1 SE above zero. A null defined by what the agents see is not exactly a null for the selection procedure built on top of it. The author keeps the 7-of-200 false detections and the Wilson upper bound of 7.0% in the text rather than rounding them to zero.
• The stylized-facts pass was fitted. ε was chosen by scanning against the checklist; many content-free processes can be tuned to fat tails and clustered volatility. The parts not calibrated against what then checks them are the phase transition and the double auction — and the phase minimum misses its published constant by a measured, unclosed gap.
• Tester Sharpe values are the tester's, at zero spread, on a synthetic symbol. Sharpe +6.79 in-sample and the reporting floor of −5.00 out-of-sample are quoted here only to show the shape of the trap; they are not comparable to any real-instrument figure, and the long-grid winner's "44 points per trade" break-even is, in the author's words, about one bar's standard deviation on that symbol — far more forgiving than a real instrument.

3 / How a reader could tear it down next

Not a signal. A next attempt with your own optimizer:

1. Build the null at 1 step per bar first, and prove it is one. Before optimizing anything, report the symbol's acf1, variance ratios at several horizons, and a split-half scan: in-sample winner's t against its out-of-sample t, over many generated markets, not one. If your aggregation already hands a fading rule a double-digit t, you have built a generator edge, not a control.
2. Keep the evidence optimizers throw away. Best and worst out-of-sample scores across the grid, the count of parameter sets profitable in-sample (3 of 20 here), and the winner's profit per trade next to a realistic spread. A winner that makes 4.4 points a trade has already told you its cost budget.
3. Run the planted-edge control. Add a known bias (the library's trend followers) at a size you choose and confirm your procedure finds it out-of-sample. A detector that has never detected anything planted cannot be trusted when it detects nothing real — and expect small planted edges (the 20-follower level) to be missed most of the time even when present.
4. Hold the three fences explicitly. Write down, before the run: rule class (lookback range and directions), horizon, and steps-per-bar. A result is a statement about that triple. Widening any one of them afterwards is a new experiment, and the long-grid result here shows why.
5. Then, and only then, the same search on real data — with costs in the symbol, a deflated correction for the grid you searched (Deflated Sharpe), and the null-market false-discovery rate (3.5%, Wilson up to 7.0% here) as the prior for what a PF of 1.12 is worth.

4 / Verdict

A useful object with an honestly fenced claim. On a market with nothing planted, for rules shorter than the agents' memory, at one step per bar, a real optimizer still hands you PF 1.12 — and it becomes −53.86 on the next bars and would not survive half a pip of spread. That measurement, reproducible on your own machine and cross-checked bit-for-bit from outside the terminal, is what the tool is for. It does not certify any strategy; it prices one failure mode of your search, provided you respect the fences the author measured rather than the marketing version ("test on synthetic data") that ignores them. The open possibility is concrete: run your actual grid, with your actual costs, through the null first, and let the null's detection rate — not the winner's equity curve — set the bar the real-data result has to clear.

Source

• Hammad Dilber, Agent Based Market Simulation: Growing a Market From Rules, and Using It to Catch an Overfitted Strategy, MQL5, 16 Oct 2026. https://www.mql5.com/en/articles/24664
• Fetched in full 2026-10-17. Attached MQL5.zip (45.96 KB per the page) was not vendored, compiled, or run here; the 82/83-pass lines are the author's printed outputs, quoted.

Numbers

From the article, not from a Quant for Free run: phase minima α 0.5111/0.4313/0.4324 (m 6/7/8) vs published 0.3374, fine-scan band 0.4113–0.5501 (N 117–1,975), collapse at α=10 volatility 0.849–0.878 for N 27–1,639; stylized facts (40,000 steps) grand-canonical excess kurtosis +0.7214, |r| acf mean 0.0752, Hill 3.97 vs control 6.17, active 97.6/1,000, ε scan kurtosis +28.666@0.0006 → +0.246@0.0024; zero-intelligence efficiency 0.9597 constrained / −0.0228 unconstrained, 29.9/60 loss-making trades; detector (200 markets) null tIn 1.672 vs tOut 0.205±0.066 (SD 0.932), detection 3.5% (Wilson 1.7–7.0%), 20 followers 18.0%, 40/80 followers tOut 31.5/47.5 at 100%; long-horizon momentum t −2.74@20 → −5.92@800, variance ratio 0.153@500 (plain); aggregation plain 30 steps/bar tOut +22.66 vs 1 step/bar +0.31; tester short grid winner +$42.54 PF 1.12 (958 trades, 4.4 pts/trade, 5-pt spread = 47.90) → −$53.86 PF 0.88 out-of-sample, 3/20 sets profitable in-sample; long grid winner +$230.52 → +$139.68 PF 2.59 (44 pts/trade); self-test 82 passed, Python cross-check 83 passed, max stats-layer rel. diff 9.6e-13. No site numbers; no summary.json (no clean-room run).