Implementation Risk — When Backtest Engines Disagree on Costs
Same strategy, same prices, same fee schedule — different open-source engines. Yin et al. (arXiv:2603.20319) isolate that gap and call it implementation risk. Numbers below are from their paper only; this page does not run a new backtest.
- What implementation risk means when strategy, data, and cost spec are held fixed
- Why zero-cost agreement is the paper’s identification trick (max pairwise difference
0.0000%) - Four metrology-style metrics: ES, IUI, DAF, CSI
- Table 3 tiers: simple ≈ 0.18% max, signal/ML 0.28–0.49%, rotation up to 3.71%
- How this pairs with cost tools, OOS checks, DSR, and walk-forward practice
The question the paper asks
Nearly every quant study leans on a backtest. Yin, Miki, Lesnichenko & Gural (2026) ask something quieter and more awkward: does it matter which simulation engine you used?
They define implementation risk as variability in backtest outcomes attributable solely to the choice of engine, with strategy logic, input data, and cost specification held fixed. In their notation (Eq. 1), if f is a summary statistic (Sharpe, total return, drawdown, …) of engine returns Re(s, d, c), then
When IR = 0 for every statistic they care about, the engines are implementation-equivalent for that (s, d, c) triple. The rest of the paper measures how far real engines sit from that ideal once costs are turned on.
Design in one paragraph
They run 15 benchmark strategies through five retained engines on 30 non-overlapping stratified buckets drawn from 180 S&P 500 constituents, under four proportional cost regimes: 0, 18, 36, and 60 basis points. The evaluation window they emphasise is 2020–2024 (full download January 2018–December 2024, 1,761 trading days, with 2018–2019 as look-back buffer for feature-heavy strategies).
Retained engines (Table 2): their reference implementation (“Ours”), bt 1.1.0, vectorbt 0.26.2, Backtrader 1.9.78, and cvxportfolio 1.3.0. Excluded after quality control: Zipline-Reloaded (calendar alignment), NautilusTrader (four compounding errors), QuantConnect Lean (no native Python API for their harness).
Benchmarks (Table 1) span five categories: simple allocation, signal-driven, ML signals, high-turnover rotation, and a zero-cost ablation (BM09 daily binary switch at 0 bps).
Four metrics (Section 3)
The paper adapts tools from metrology / sensitivity analysis rather than inventing new statistics from scratch:
Engine spread (ES)
For statistic values {f1, …, fK} across K engines, they define a CV form and a range form (Eqs. 2–3). Primary reporting uses the range:
They prefer the range form because the CV form is unstable when the cross-engine mean is near zero.
Implementation uncertainty interval (IUI)
With K = 5, they use t0.025, 4 ≈ 2.78. Unlike a GUM-style standard error of the mean, IUI uses sf directly so the band characterises the full engine spread, not only uncertainty on the mean.
Divergence amplification factor (DAF)
Reference = BM01 (equal-weight monthly), with ESrange = 0.379 percentage points. DAF > 1 means the benchmark amplifies cost-model disagreement beyond that baseline.
Conclusion sensitivity indicator (CSI)
CSI = 1 is the paper’s red flag: go/no-go depends on which engine you happened to use.
Result 1 — zero cost, perfect agreement
BM09 (daily binary switch, 0 bps) is the causal control. All 10 pairwise comparisons yield a relative difference of exactly 0.0000% in total return. When the cost model is removed, the five engines agree on rebalancing calendar, price look-up, and return compounding. Any divergence under nonzero costs can therefore be attributed to cost-model implementation — not to a broken harness.
Result 2 — cost-driven divergence (Table 3)
Table 3 reports mean and maximum pairwise relative difference in total return (%) across the five engines (10 pairs), averaged over 30 buckets. Three tiers appear:
| Tier (paper) | Benchmarks | Max pairwise (paper) |
|---|---|---|
| Simple / ablation | BM01, BM02, BM06, BM09, BM12 | ≤ 0.1799% (BM09: 0.0000%) |
| Signal / ML | BM05, BM07, BM08 variants | 0.2750% – 0.4881% |
| Rotation | BM03, BM04, BM10, BM11 | up to 3.7077% (BM04 @ 36 bps) |
Source: Yin et al., Table 3 (pairwise relative difference in total return). Abstract rounds the rotation extreme to 3.71%.
Headline abstract numbers (same paper, verified against Table 3 / §5):
- Divergence scales with cost intensity: Spearman ρ = 0.93, p < 0.001
- Remains below 0.75 percentage points for 12 of 15 benchmarks
- Reaches 3.71% in total return for high-turnover rotation under the heaviest regimes — framed in the abstract as ambiguity equivalent to roughly $37 M per year for a $1 B portfolio
Result 3 — ES, IUI, DAF, CSI (§5.4)
| Benchmark | ES (pp) | IUI width (pp) | DAF |
|---|---|---|---|
| BM01 (reference) | 0.379 | 1.069 | 1 (by construction) |
| BM11 | 0.874 | 2.408 | 2.308 |
| BM04 | 1.040 | 2.079 | 2.747 |
| BM03 | 2.339 | 5.942 | 6.178 |
Source: Yin et al. §5.4.1–5.4.3. Rotation strategies amplify cost-model disagreement by a factor of about two to six relative to BM01.
CSI = 0 across all 15 benchmarks
Engine choice never reverses the sign of the Sharpe ratio in their sample. For BM11 (negative mean Sharpe), all five engines agree on unprofitability — e.g. ours / vectorbt = −0.1151, bt = −0.1219, backtrader = −0.1225, cvxportfolio = −0.1211. Closest gap: 0.0073 in absolute Sharpe between −0.1225 and −0.1151. Cardinal numbers move; the go/no-go sign does not, for these strategies.
Forensics — seven defects, five failure modes
Quality-control forensics on engines examined during the study uncover seven previously undocumented defects spanning five categories: cost-model bugs, infrastructure bugs, architectural bugs, specification divergence, and complexity sensitivity.
The abstract’s flagship example is Backtrader’s default: with percabs=False, the engine divides the user-supplied commission by 100 before applying it. Passing commission=0.0018 intending 18 bps yields only 0.18 bps — a silent factor-of-100 under-charge. The paper’s fix for their harness is percabs=True (plus a sells-first fill-ordering patch for a separate infrastructure bug).
How this fits Quant for Free
Costs are not a footnote. Before you trust a pretty equity curve, stress the cost model the way this paper stresses the engine:
- Vary fee assumptions in the backtest costs tool
- Keep an honest out-of-sample habit with OOS Lab
- Deflate search-inflated Sharpes via the Deflated Sharpe Ratio lesson
- Structure walk-forward work with the walk-forward Python guide
Implementation risk and statistical overfitting are independent failure modes: a strategy can clear every multiple-testing gate and still print different totals depending on which library applied the commission.
- Implementation risk
- Outcome variability from engine choice alone (strategy, data, cost spec fixed).
- ES
- Engine spread — primarily max − min of a statistic across engines.
- IUI
- Implementation uncertainty interval — f̄ ± t · sf band across engines.
- DAF
- Divergence amplification vs BM01’s ESrange = 0.379 pp.
- CSI
- 1 if Sharpe signs disagree across engines; 0 otherwise (paper: CSI = 0 for all 15).
- Zero-cost ablation
- BM09 — proves harness identity when costs are off.
Where to go next
ToolBacktest costs — see how fee assumptions move the curve Tool
OOS Lab — hold-out structure before you over-trust in-sample Learn · Deep dive
The Deflated Sharpe Ratio — selection bias under multiple testing Guide
Walk-forward validation in Python — process, not a single backtest