Quant for Free
Method pointer

Nested CV: V-in-V evaluation without reopening the test set

UnifiedValidationCalibrator is the production class from the AFML MetaTrader 5 blueprint (Part 16): three temporal zones, nested walk-forward/CPCV loops, Masters’ 1-SE hyperparameter rule, out-of-fold isotonic calibration, and a final test gate that raises if you open it twice. Python selects and calibrates; MQL5 consumes the artifacts.

ridingyo · 2026-09-20 · source mql5.com/en/articles/22040 (Patrick Murimi Njoroge, 20 May 2026)

What the article ships

AFML Part 16

Attached code in afml.zip: nested_cv.py (DataPartition, partition_data, inner_cv_search, UnifiedValidationCalibrator), plus dependencies cross_validation.py and combinatorial.py.

from afml.cross_validation.nested_cv import UnifiedValidationCalibrator
V-in-V 1-SE rule OOF isotonic sklearn-compatible
Open MQL5 article → oos-lab toolkit

Nested CV vs oos-lab vs DSR

choose
  • Use Nested CV (UnifiedValidationCalibrator) You are fitting a classifier, searching a param_grid, and learning a probability calibrator under temporal labels (t1, embargo). You need an estimate that is not contaminated by HP selection or in-sample calibration — then open the final ~20% once.
  • Use oos-lab You already have return series (or a matrix of variants) and need PSR / Deflated Sharpe / PBO–CSCV / WalkForward or CombinatorialPurgedKFold index splits / Harvey–Liu haircuts in Python — not model selection + calibration orchestration.
  • Use the Deflated Sharpe calculator Quick browser check from an annualized Sharpe, observation count, and trial count. No install. Complements nested CV after you have a committed OOS return path — it does not replace nested CV.

What the pipeline actually does

Documented from the Part 16 article only — not invented. Three leakage paths are blocked by three architectural layers:

Zones

60 / 20 / 20 V-in-V

Outer training (~60%): all nested CV, search, OOF, calibration. Inner validation (~20%): shortlist checkpoint — inspect, do not retune. Final test (~20%): open exactly once.

Selection

Inner loop + 1-SE

Anchored expanding PurgedWalkForwardCV. Among configs within one SE of the best mean score, pick the simplest (first in ParameterGrid order).

Calibration

OOF then gate

Second inner walk-forward pass builds OOF probs; isotonic fits on those. Consensus params by majority vote. Final test returns raw/calibrated Brier and log loss.

partition_data

inner_val_pct=0.20, final_test_pct=0.20

Temporal iloc split into DataPartition. Preserves index alignment with t1 for purged CV.

outer_cv_type

'walkforward' | 'cpcv'

Validated at construction. CPCV mode needs close_prices and primary_sides on fit(); exposes cpcv_distribution_metrics_ after fit.

_oof_for_fold

fresh PurgedWalkForwardCV + clone

After 1-SE selection, refits per inner split and writes predict_proba[:, 1] into a full-length OOF series for the fold calibrator.

evaluate_final_test

single open via _final_opened

Second call → RuntimeError. Metrics named in the article: raw & calibrated Brier, raw & calibrated log loss, plus probability arrays.

Constructor / fit contract (from §7)

__init__ stores scalars and the estimator only. Data-carrying arguments — t1, sample_weight, close_prices, primary_sides — belong to fit(). Example kwargs shown in the article: n_outer_splits, n_inner_splits, pct_embargo, min_train_size, scoring='neg_brier'; for CPCV also cpcv_n_folds, cpcv_n_test_folds.

Usage snippet (from the article)

Illustrative estimator and grid only — not a strategy result. After attaching the AFML modules from the article zip:

from sklearn.ensemble import RandomForestClassifier
from afml.cross_validation.nested_cv import UnifiedValidationCalibrator

uvc = UnifiedValidationCalibrator(
    estimator=RandomForestClassifier(random_state=42),
    param_grid={
        'n_estimators': [100, 200, 500],
        'max_depth': [3, 5, 8],
    },
    outer_cv_type='walkforward',
    n_outer_splits=5,
    n_inner_splits=3,
    pct_embargo=0.01,
    min_train_size=0.1,
    scoring='neg_brier',
)

# Run the full nested CV pipeline
uvc.fit(X, y, t1=events['t1'], sample_weight=events['tW'])

print(uvc.outer_scores_summary())

# Open the final test set — EXACTLY ONCE
result = uvc.evaluate_final_test()
print(f"Final Brier (cal): {result['brier_cal']:.4f}")

probs = uvc.predict_proba(X_new)[:, 1]

Order each param_grid list simple → complex so the 1-SE “first within band” pick is meaningfully the simplest. The article’s cost example: 5 outer × (9 × 3 + 3) = 150 model fits; CPCV with N=6, k=2 → 15 outer splits → 450 fits. Rough data guideline: ≥ 2,000 independent observations after accounting for label concurrency.

When not to use nested CV

Per the article’s Practical Considerations: nested CV is for deploying with the hyperparameters and calibrator it selects. It is unnecessary when hyperparameters are fixed externally — then a single-loop OOF calibration via CalibratorCV (Part 12) is enough. It is also unnecessary for exploratory research that does not need unbiased performance estimates; the compute cost slows iteration without corresponding benefit.

Do not wrap UnifiedValidationCalibrator inside another meta-estimator such as GridSearchCV or CalibratedClassifierCV — the article calls that a conceptual error (nested CV inside nested CV with no principled separation).

Where this sits on Quant for Free

oos-lab is the stats/splitter toolkit for returns you already have. The Deflated Sharpe calculator is the browser front door to selection-bias adjustment on a single Sharpe. Nested CV sits earlier in the research stack: honest model selection and calibration before you trust an OOS return series enough to feed those tools. Pair with Module 5 — Validation Gauntlet and the walk-forward guide (Walk-Forward Optimization in Pure Python).

Tools

oos-lab

PSR, DSR, PBO/CSCV, WalkForward & CombinatorialPurgedKFold — MIT Python validation stats.
Tools

Deflated Sharpe calculator

Browser DSR / PSR from a Sharpe, observation count, and trial count.
Learn

Validation Gauntlet

Course module that sequences the honest checks a backtest has to clear.
Guides

Walk-forward in Pure Python

Rolling train/test WFO patterns that pair with purged walk-forward inner/outer loops.

Questions

What is UnifiedValidationCalibrator?
A sklearn-compatible Python class from MQL5 article 22040 (AFML Blueprint Part 16). It runs nested CV with a three-zone V-in-V partition, Masters’ 1-SE hyperparameter rule, OOF isotonic calibration, consensus params by majority vote, and evaluate_final_test() gated to open once. Methods shown: fit, predict, predict_proba, outer_scores_summary, evaluate_final_test.
When should I use Nested CV vs oos-lab vs the Deflated Sharpe calculator?
Nested CV when selecting HPs and fitting a calibrator under temporal leakage constraints. oos-lab when you need PSR/DSR/PBO/haircut/splitters on return series. The DSR calculator for a fast single-figure browser check. They stack; they do not substitute for each other.
What does the final test report?
Per §6 of the article: raw and calibrated Brier score, raw and calibrated log loss, plus the raw and calibrated probability arrays for downstream analysis.
Is this investment advice?
No. Educational and methodological use only. This page points at a published validation procedure; it does not recommend any security, strategy, or trade, and makes no claim about future results.
Educational tool, not investment advice. This page summarizes an open MQL5 article’s nested-CV design and does not predict performance or recommend any trade. Simulated and backtested results have inherent limitations. Verify any figure independently. Past results, real or simulated, do not guarantee future outcomes. See the disclaimer.