Where Backtest and Live Part Ways — Six Defects and a Ten-Point Checklist

Most writing on backtest failure is about statistics: too many trials, a winner picked from noise. Alex Vidovich and Zivana Zerjal's Where Backtests and Live Trading Part Ways (arXiv:2610.21846, submitted 15 Oct 2026; working paper dated 24 Sep 2026) is about the layer underneath the statistics. One intraday US-equity system, run live and replayed day by day against its own backtest, produced six defects — three data, two code, one configuration — none of which showed in summary statistics, and all of which sat where two paths that should agree had drifted apart. The authors make no claim about whether the system is profitable, and neither does this guide. It is a how-to for anyone moving a vectorbt/Optuna research path toward a live runner: the transferable unit is not their magnitudes (they say explicitly that those are theirs and do not transfer) but the guard test each defect now carries.

Setting, as reported: a fixed universe of 97–98 US stocks, 1-minute bars, decision shortly after the open, run on paper and with real money; market data from Alpaca's API through one fetch function serving both the backtest cache builder and the live runner. The main backtest (the parity harness) drives the same strategy code as live and is checked against live fills. Parity standard used in the paper: on matched real-money days, replay and live results for the same trade may differ by 5% of the replay result or a small fixed dollar floor, whichever is larger (the floor's size is withheld). Five real-money days were replayed: two matched and passed that bar, one diverged and was later traced to Defect 5, and a later apparent pass turned out to be two errors cancelling. Matching decisions is necessary, the authors conclude, not sufficient.

1 / What the object actually is

Each defect is reported in four parts — symptom, magnitude, root cause, detection test. Magnitudes below are the authors' log figures, quoted; some are rounded or withheld in the paper to protect the live system, and the underlying research log is not public.

Defect 1 — the default feed is not the consolidated tape (data). A historical request carrying no feed parameter returned single-exchange (IEX) data instead of the consolidated tape (SIP) in the authors' test. Cache opening-minute volume was a small fraction of live's: their Table 1 gives one cached 9:30 bar at ~7,900 shares vs ~3.4 million consolidated (0.2%), another at 0.5%, and a 9:32 bar at 1.4%; the log's range is given twice and the paper uses the later, wider one, 0.1–2% of true opening-minute volume. Full-day totals were correct — only the per-minute split at the open was thin — and live was unaffected. A scan of all 98 cached 2026 trading days found at least one liquid name with a 9:30 bar under 50,000 shares on every day. On two months replayed both ways the same trades were chosen, but the thin cache understated one month and overstated the other, turning one losing trade into a winning one; an earlier permutation test on the thin cache (p = 0.217) was marked invalid and retired. Guards: a request-recording test asserting every request pins the consolidated feed, plus a runtime alarm if opening volume collapses across nearly the whole universe at once at decision time (it logs and alerts; it never blocks a trade).

Defect 2 — dividend-adjusted history against unadjusted live fills (data). A commit switched the fetch's adjustment setting from split-only to "all" three and a half weeks before discovery. One rebuilt bar sat −0.68% below the live price basis at identical volume; a same-day scan found 37 of 98 symbols shifted the same way, every one a dividend payer, log examples from −0.68% to −1.65%. The replay combined an unadjusted entry with an adjusted exit; after re-pulling unadjusted prices, entry and exit matched live to the cent. Guard: the same request-recording pattern asserting unadjusted ("raw") prices — and, what actually found it, the parity step: before trusting a rebuilt dataset, replay one live trade and match entry and exit to the real fill.

Defect 3 — a sizing step that scaled past the binding limit (code). Sizing is a chain. The strategy applied the buying-power limit correctly; a later step that can increase size multiplied the already-limited notional with no second limit, reading a field that exists only in the backtest. In the reference backtest, 20 of 61 simulated fills were sized at a notional the live account can never take; the log estimated that backtest's dollar figures at about 18% too large in absolute size and ordered every earlier dollar figure discarded. The fix changed notionals, not which trades were taken (61 fills before and after). It was the second defect of its shape in nine days, and a third instance appeared later. Guard: a unit test that builds a setup already at its limit, applies the size-increasing step, and asserts ceiling, notional and share count are unchanged; structurally, each setup now carries its limit so any later step can re-apply it.

Defect 4 — the clock is UTC, the market is New York (code). Research scans picked the opening bar with the fixed key "13:30" against cache timestamps stored in UTC. 13:30 UTC is 9:30 ET only in daylight time; in winter it is 8:30 ET, an hour before the open. The study period had 129 winter days, 30% of the sample; on a 13-day winter sample, 16% of stock-mornings were scored on a pre-market bar and 84% silently dropped by a completeness check — so the scans described about 70% of the period plus a pre-market slice and reported it as the whole. Eleven scans were affected and re-run the same morning; in the corrected table no verdict flipped and almost every number moved (one shown study grows from 2,207 to 3,024 stock-mornings). A second form — building the hour as 13 + minutes // 60, hard-coding the summer offset — briefly printed a positive result that vanished with the fix and was never quoted. The same trap had been logged months earlier as an operational note, not turned into shared code, so it came back. At paper-writing time there was still no unit test for this; the control is one shared exchange-time helper plus a written rule that a bare string slice of a cache timestamp is a bug on sight. The missing test the authors name: one winter date and one summer date, asserting 9:30 ET resolves to the right bar on both.

Defect 5 — data the live path downloads and the replay did not (data). Live fetched data beyond the stocks it trades; the replay built its request from the traded list. The two lists had once matched and drifted apart. With the extra data missing, an input feeding several decision rules silently read zero (or was partly computed) in every harness run for about four months. Restoring it, the replay reproduced live's value exactly and made the same decision within the slippage bar; across 20 real-money mornings (replay before vs after the fix, not scored against live) it changed one decision in twenty. Every harness figure from those four months carries the defect, size and direction unknown, none re-run yet. Guards: three tests written first and failed for the right reasons — the harness now warns and records the gap in every day's output when a required input is missing, builds its request from the live path's own list, and asserts the extra data is in the request. The gap can no longer be silent.

Defect 6 — "enabled in config" is not "connected in the execution path" (configuration). A family; the paper shows three of six logged members. (i) A new exit rule was on, but its reason was not in the live monitor's honored set — live ignored it, and the simulation ending at the ignored exit hid every later honored exit. (ii) A sizing rule was wired only into a backtest-only branch; two live trades were sized as if it did not exist, and its "end-to-end confirmation" had checked that the flag loaded into config, not that orders changed — a planned upgrade armed into the same dead branch would have done nothing. (iii) An audit found 53 keys read by production code but undeclared in config, about a quarter of the surface, one of them on by default; the service's startup file carried five stale settings of its own, including the switch that turns order execution on, which existed only there — shipping the generated defaults would have silently put the live system into dry-run. One regression map was itself false assurance: it listed four keys the runtime never reads, so its test passed while guarding nothing. Guards, in three layers: a mapping test (every exit flag set to on maps to a reason in the live monitor's honored set, and every mapped key is really read by production code), a presence test (every key the code reads is declared), a neutrality test (loading declared values reproduces code defaults apart from documented exceptions), and a preflight script running twice a day on the live server that checks behaviour in logs and output files, not settings.

The ten-point checklist (paper's Table 4, traced to defects): 1 Pin the data feed in every request; test the parameter. 2 At decision time, alarm if opening volume collapses across the whole universe at once. 3 Use unadjusted prices for execution simulation; test the parameter. 4 Replay live trades regularly and match decisions, prices and sizes; after any data change, match one fill to the cent. 5 Re-apply the binding size limit after every step that can increase size; test at the limit. 6 Convert timestamps in one shared exchange-time helper, tested on a winter and a summer day. 7 Build the replay's data request from the live path's list of required inputs; make any missing input loud, never a silent default. 8 Chase every unexplained difference between replay and live to its root cause. 9 For every setting, test that it changes an observable action in the order path; declare every key the code reads. 10 Check live behaviour in logs daily, not settings.

2 / What it does not claim — and where it fails

• One system, one vendor, one account. The authors state it first among their limitations: magnitudes, and dollar sizes in particular, are theirs and do not transfer; vendor behaviour is what they observed on their subscription in 2026 and may differ or change. What transfers is the defect classes — and they have not shown how often those occur elsewhere.
• One clear instance per class was chosen. Larger errors that are not among the six are mentioned (a retired second harness that returned less than half the parity figure at the same size limit because it misread its inputs, yet had matched live decisions on 7 of 8 real-money dates). The log does not count its defects; this is not a prevalence sample.
• Several headline checks were run while Defect 5 was present. The paper flags this repeatedly: the 5-of-5 earlier match, the two-month Defect 1 comparison, the Defect 3 dollar estimate. Fixing Defect 5 also exposed that live and replay applied a size cap at different points, and on one real-money day two errors cancelled into a false pass. Treat every figure from a harness later found defective as provisional — the authors do, including their research graveyard of rejected ideas, every rejection recorded while Defect 5 was present being provisional until re-run.
• Written lessons did not stop recurrence. The clock trap was logged and returned months later; the sizing-chain lesson returned as well; Defect 3's test was in place and its class came back in a step the test did not cover. The paper's own data point is that tests alone are not enough — which is why checklist items 8 and 10 are habits (chase every divergence; check behaviour in logs daily), not assertions.
• The parity harness itself has a known gap. Live-only execution checks that need live data are not replayed; per-trade dollars differ even on matched days because the harness uses a cached spread and live uses the real quote. The authors trust decisions and aggregates more than single trades, and only as far as that evidence reaches. Every parity day so far also fell in US summer time.
• Unverifiable here. Figures come from the authors' dated research log, which is not public; some are rounded or withheld (including the slippage floor). Two log entries disagree on Defect 1's range and the paper adopts the wider one — this draft follows the paper, but there is no underlying data for a reader to audit. If a claim matters for your stack, the guard test is the part to copy, not the number.

3 / How a reader could tear it down next

Not a signal. A parity pass on your own vectorbt/Optuna → live path, in the paper's order of operations:

1. Replay one live (or paper) day before you trust any rebuilt dataset, and match one fill to the cent — decision, entry, exit, size. This single habit is what found Defect 2 and exposed Defect 5; summary statistics found none of the six.
2. Record your requests. Wrap the data fetch in a fake/recording session in tests and assert the parameters that change what the data means: feed, adjustment basis, session, timezone handling. A default that looks right today is Defect 1 waiting for a vendor change.
3. Make missing inputs loud. Build the replay's data request from the live path's required-inputs list, and fail (or at minimum stamp every output) when an input is absent. A silent neutral default — zero — ran for four months in this system.
4. Re-apply the binding limit last. Enumerate every step that can increase size and test at the limit with a position already there. If a sizing field exists only in the backtest, assume the chain diverges until a test proves otherwise.
5. One exchange-time helper, two test dates. No bare string slices of timestamps anywhere; a winter and a summer date in every time-dependent test. Then grep your research scans for hard-coded offsets — the second form of Defect 4 lived in scan scripts, not the harness.
6. Test that each setting changes an observable action in the order path. Declare every key the code reads, generate the declaration from code defaults, and diff the result against every place settings can live (config file, startup file, environment). Then check behaviour in production logs daily — the config file may not be the only place settings live, and in this system the execution switch lived only in the startup file.
7. Keep a graveyard, and re-run it when the harness changes. One entry per rejected idea with the test that rejected it — and mark every rejection provisional when a harness defect is found, because a rejection is only as good as the harness that made it.

4 / Verdict

None of the six was found by reviewing summary results; each was found by a narrow, concrete look — one live trade against the cache, one bar from four sources, one day's decision record re-read, one unexplained real-money day chased to the end. A bug that makes results absurd gets caught; these made results look plausible, erred in both directions, and in Defect 4's case moved almost every number while flipping no verdict. The checklist is therefore not bureaucracy: it is the six narrow looks, turned into tests and two daily habits. A shippable edge is still rare, and this paper does not move that bar — it protects the bar from being measured with a broken ruler. The next attempt it leaves open is unglamorous and specific: pick the checklist item your stack has never done (for most research-to-live paths it is item 4 or item 9), implement its guard test first, and let it fail for the right reason once.

Source

• Alex Vidovich & Zivana Zerjal, Where Backtests and Live Trading Part Ways: Six Defects from One Intraday System, arXiv:2610.21846 [q-fin.ST], submitted 15 Oct 2026 (working paper 24 Sep 2026; also on SSRN, DOI 10.2139/ssrn.7517378). https://arxiv.org/abs/2610.21846
• Abstract page and full PDF fetched and read 2026-10-17. The authors' research log is not public; no code or data accompanies the paper that we could run, and nothing was run here.

Numbers

From the paper, not from a Quant for Free run: universe 97–98 stocks; slippage bar 5% of replay result or a withheld dollar floor; Defect 1 ranges 0.1–0.5% vs 0.1–2% (paper uses the wider), Table 1 points 0.2% / 0.5% / 1.4%, 98 cached days each with ≥1 liquid 9:30 bar <50,000 shares, retired permutation p = 0.217; Defect 2 bar −0.68%, 37/98 symbols, range −0.68% to −1.65%, adjustment changed ~3.5 weeks before discovery; Defect 3 20/61 fills oversized, dollars ≈18% too large, 61 fills before/after; Defect 4 129 winter days (30%), 16% pre-market / 84% dropped on a 13-day winter sample, ≈70% coverage, 11 scans, Study A 2,207 → 3,024 stock-mornings; Defect 5 ≈4 months, 1 decision changed in 20 real-money mornings; Defect 6 53 undeclared keys (≈a quarter of the surface), 5 stale startup settings, 4 never-read keys in one regression map; retired harness <half the parity figure at the same limit, 7/8 decision match. No profitability figures are reported in the paper, and none are reported here. No site numbers; no summary.json (not a backtest).