Skill or luck? Test it.

Strategy evaluation is a statistical problem

A backtest result is not a fact about a strategy. It is one draw from a random process — one path the world happened to take, scored once. Evaluation means asking about the process: if the strategy has no real edge, how often does a result this good appear anyway?

That reframing sounds academic until you count what it protects you from. Take a strategy with an annualized Sharpe of 1.2 over ten years. Three separate mechanisms hand out numbers like that for free. The market drifts — stocks rose for most of the last century, so any rule that ends up long more often than not inherits a positive Sharpe it did nothing to earn. Diversification smooths — spread a coin-flipping allocator across nine sector ETFs and its equity curve looks respectable, because averaging across assets hides the coin. Selection picks winners — if you tried twenty parameter settings and kept the best, the number you are looking at is the best of twenty draws, which is large even when all twenty are worthless. None of these mechanisms is visible in a tear sheet. All of them are visible to statistics.

Two engineering preconditions make statistical evaluation possible at all, and the engine treats both as hard requirements. Determinism: every random element — permutation order, bootstrap resamples, optimizer sampling — consumes an explicit seed, so the same run produces the same p-value and the same SHA-256 fingerprint, today and in CI next month. A significance number you cannot reproduce is an anecdote. Honest accounting: decide-at-t-earn-at-t+1 discipline, costs applied identically to whatever is being compared, and NaN where data is missing rather than a plausible-looking zero. Statistics computed on top of look-ahead or fabricated fills are precise nonsense.

The toolkit then stacks into layers, each answering one question about the process:

Start with the error bars: how noisy is a Sharpe ratio?

Before any hypothesis test, it pays to know how blurry the measurement itself is. The answer is: much blurrier than intuition suggests.

Fix the notation first, symbol by symbol. The Sharpe estimate is the mean excess return over its standard deviation: μ̂ is the average per-period (daily) return, r_f the per-period risk-free rate, σ̂ the standard deviation of the same returns. Annualizing multiplies the daily ratio by √252:

Every symbol on this page keeps these meanings: hats mark quantities estimated from data (and therefore noisy), T counts the observations behind the estimate, B counts the resamples or permutations we generate.

Under textbook assumptions — independent, identically distributed, roughly normal returns — the standard error of a Sharpe estimate has a closed form (Lo, 2002):

Plug in ten years of daily data (T = 2520) and a true Sharpe of zero: the annualized estimate still wobbles with a standard deviation of about 0.32. A strategy with no edge whatsoever shows Sharpe above 0.5 roughly six percent of the time — before any selection effects. And this is the optimistic case: daily returns have fat tails, they cluster, and a strategy's returns are a mixture of in-market and in-cash days. Every violated assumption widens the true spread beyond the formula — always in the direction that flatters you.

Because the closed form under-reports the noise, the engine estimates the uncertainty from the data instead, with a stationary block bootstrap: resample the return series in wrap-around blocks of random, geometrically distributed length (expected √T), recompute the Sharpe on each resample, and read the 5%/95% quantiles. Blocks — rather than single days — preserve the short-range autocorrelation that an i.i.d. bootstrap would destroy:

from backtester.walkforward.statistics import bootstrap_sharpe_ci

# Stationary block bootstrap (Politis & Romano 1994): resamples
# wrap-around blocks of expected length sqrt(T), preserving the
# short-range autocorrelation an i.i.d. bootstrap would destroy.
ci = bootstrap_sharpe_ci(
    daily_returns,            # the strategy's return series
    n_resamples=1000,
    seed=42,                  # WalkForwardConfig.seed for reproducibility
    risk_free_rate=0.0,
    trading_days=252,
)
lo, hi = ci                   # 5% / 95% bounds on the ANNUALIZED Sharpe
print(f"Sharpe 90% CI: [{lo:.2f}, {hi:.2f}]")

# Rule of thumb: a CI whose lower bound hugs zero is a strategy whose
# entire track record is compatible with "no edge".

The interval is a reality check, not a verdict: it tells you the measurement's blur, but it cannot say whether the point estimate is skill. For that you need a reference distribution for "no skill" — and that is what permutation builds.

The idea: shuffle and compare

Forget formulas for a moment. Here is the entire method in three sentences.

Your strategy made a sequence of decisions: long on some days, flat on others. If those decisions carry real information about what the market does next, their timing matters — being long on the 14th and flat on the 15th was the point. And if the timing matters, the real arrangement should beat the same decisions dealt out in random order.

So: move the strategy's decision path against the same market and re-score it. For persistent or cost-aware strategies, the primary intervention is a non-zero circular shift of the entire weight matrix: it preserves the cyclic position sequence, holding periods and nearly all turnover while breaking calendar alignment. A thousand shifts give the null distribution for "this decision path without its original calendar alignment." Full random row permutation answers a more aggressive question because it also destroys serial persistence and turnover structure.

A six-day example you can check by hand

Before touching real data, the entire procedure on six trading days — small enough that every number is verifiable with mental arithmetic.

One instrument. Six days of returns. The strategy decides each evening whether to be long the next day (weight 1) or stay in cash (weight 0). Its five decisions look clairvoyant — in the market for all three up days, in cash for both down days:

+1.8% while the market itself made +0.1%. Skill? Ask the shuffles. The decision sequence is {no, yes, no, yes, yes} — three "yes" days out of five. A shuffle keeps the same three yes-days but deals them to random dates. There are only ten possible arrangements; four draws:

Zero of four shuffles matched the real strategy. The permutation p-value counts exactly this, with a +1 in both places so an honest test can never report a probability of zero:

Read it out loud: "if the timing carried no information, I would see a result this good about one time in five." Not impressive — and that is the honest verdict from four shuffles, because with B = 4 the best possible p-value is 0.20. Real runs use B = 1000, where the finest verdict becomes ~0.001. The whole example is ten lines of NumPy, worth running once to see the machinery breathe:

import numpy as np

r = np.array([-0.5, 0.8, -1.2, 0.6, 0.4])   # market returns, days 2..6 (%)
w = np.array([0, 1, 0, 1, 1])               # decisions held on those days

observed = (w * r).sum()                     # +1.8%

rng = np.random.default_rng(42)
B = 10_000
null = np.array([(rng.permutation(w) * r).sum() for _ in range(B)])

p = (1 + (null >= observed).sum()) / (B + 1)
print(f"observed {observed:+.1f}%  |  p = {p:.3f}")
# observed +1.8%  |  p = 0.099
# (exact: 1/10 arrangements ties the maximum - see the table above)

With all 10,000 shuffles the p-value settles near 0.099 — because exactly 1 of the 10 possible arrangements (the real one) achieves +1.8%. Everything the engine does is this exercise, scaled up: more days, more instruments, Sharpe instead of total return, and the bookkeeping that keeps the comparison honest.

Choose the null by what it preserves

The art of a permutation test is choosing what to destroy and what to preserve. Destroy too little and the null still contains skill — the test is blind. Preserve too little and the null is a strawman every strategy beats. The engine therefore exposes several nulls, each engineered around a different invariance.

Circular-shift the decision path (circular_shift_weight_rows): the entire target-weight matrix moves by one non-zero offset against returns. The cyclic sequence of positions, holding periods and transitions survives; only calendar alignment changes. This is the primary timing null. A block permutation preserves order inside contiguous blocks and is a useful secondary specification.

Full row permutation (permute_weight_rows) preserves each row and the cross-sectional distribution of weights, but it also destroys serial persistence, holding periods, exposure autocorrelation, turnover structure and regime association. It is an aggressive sensitivity test, not a pure timing intervention. That distinction is especially important when costs are enabled: the same cost formula can charge a row-permuted null far more turnover than a persistent observed strategy.

Shuffle the market (permute_bars): think of each price series as a wall of daily bricks — one brick per day's log-return. Pull the bricks off, shuffle the pile, rebuild the wall from the same starting price. With P_t,j the price of instrument j and one shared permutation π across all instruments.

In symbols, one day's brick is the log-return — the form of return that adds over time, which is exactly why bricks can be rearranged:

and rebuilding the wall means compounding the shuffled bricks from the original first price:

One line shows why the endpoints survive every shuffle — a sum does not care about the order of its terms, so the last rebuilt price always equals the real one:

This asks the deeper question: did the rule need real temporal structure to make its money? Four invariants follow directly from the construction:

import numpy as np
import pandas as pd

def permute_bars(prices: pd.DataFrame, *, rng: np.random.Generator) -> pd.DataFrame:
    """Masters-style multivariate bar permutation (simplified core)."""
    log_prices = np.log(prices.to_numpy(dtype=float))
    log_rets = np.diff(log_prices, axis=0)          # (T-1) x N daily "bricks"

    perm = rng.permutation(len(log_rets))           # ONE shared permutation ...
    permuted_rets = log_rets[perm]                  # ... moves whole rows

    rebuilt = np.empty_like(log_prices)
    rebuilt[0] = log_prices[0]                      # anchor: same first price
    rebuilt[1:] = log_prices[0] + np.cumsum(permuted_rets, axis=0)

    return pd.DataFrame(np.exp(rebuilt), index=prices.index, columns=prices.columns)

from backtester.validation import (
    block_permute_weight_rows,
    circular_shift_weight_rows,
    permute_bars,
    permute_weight_rows,
)

rng = np.random.default_rng(42)

# Primary timing null: shift the COMPLETE decision path.
# Serial persistence, cyclic holding periods and transitions survive;
# calendar alignment to returns changes.
w_null = circular_shift_weight_rows(weights, rng=rng)

# Alternative: preserve order inside contiguous blocks.
w_block = block_permute_weight_rows(weights, rng=rng, block_size=20)

# Aggressive sensitivity only: preserves the row distribution but
# destroys persistence, holding periods and turnover structure.
w_aggressive = permute_weight_rows(weights, rng=rng)

# Scheme 2: shuffle the MARKET, keep the rule.
# One shared permutation across instruments - the correlation matrix
# and each instrument's return distribution are preserved exactly;
# trends, regimes and autocorrelation die.
p_null = permute_bars(prices, rng=rng)

# Both reject dishonest input loudly: non-finite weights, interior
# NaN price gaps, non-positive prices -> ValueError, never a guess.

The engine implementation adds the honesty details the sketch omits: instruments entering the sample late keep their ragged head in place and only fully-observed rows are shuffled; interior NaN gaps are rejected loudly rather than bridged with invented prices; and every function takes an explicit seeded generator — library code never hardcodes randomness.

Test 1 — signal timing, on real data

The six-day exercise, industrialized. Null hypothesis, in words: "the timing of my strategy's decisions carries no information about future returns."

The engine's implementation is honestly just this loop — worth reading once, because there is no magic left after it:

def portfolio_returns(W: np.ndarray, R: np.ndarray) -> np.ndarray:
    """One scoring path for the observed run AND every null draw.

    Held weights are the previous row's targets: decide at t, earn r[t+1].
    """
    held = np.vstack([np.zeros((1, W.shape[1])), W[:-1]])
    return (held * R).sum(axis=1)

# The whole test is one loop
rng = np.random.default_rng(seed)                    # seed threaded, never hardcoded
observed = sharpe(portfolio_returns(W, R))

null = np.empty(n_permutations)
for i in range(n_permutations):
    shift = rng.integers(1, len(W))                  # non-zero calendar shift
    W_null = np.roll(W, shift=shift, axis=0)         # complete path moves together
    null[i] = sharpe(portfolio_returns(W_null, R))   # same market, same scoring

p_value = (1 + (null >= observed).sum()) / (n_permutations + 1)

Two details carry the statistical honesty, and both are visible in those twelve lines:

Decide at t, earn at t+1. Held weights are the previous row's targets. Neither the real run nor any null run can peek at the bar it is scored on — the same look-ahead discipline the engine enforces everywhere else. One scoring path. The real strategy and every null are scored by the same reconstruction, with costs calculated from each path's own turnover. The engine's own NAV Sharpe is printed as a reference, but never enters the test statistic — the comparison's validity comes from both sides sharing one scoring path, not from matching execution detail.

The same three steps in symbols, one at a time. The portfolio return on day t is the sum, across all N instruments, of yesterday's target weight times today's instrument return:

Each draw circularly shifts the complete weight matrix W by a non-zero offset and re-scores it on the unchanged return matrix R — producing one null draw per shift:

The p-value ranks the observed metric m̂ inside those B draws (the add-one formula from the six-day example), and the z-score expresses the same position in units of null standard deviations — convenient for comparing strategies, while the p-value stays the decision number:

from backtester.validation import MCPTConfig, MCPTRunner

config = MCPTConfig(
    mode="signal_timing",
    n_permutations=1000,     # p-value resolution ~0.001
    metric="sharpe",          # or "total_return" / "profit_factor"
    cost_bps=None,            # optional cost from each path's own turnover
    weight_null="circular_shift",  # primary; or block_permutation / row_permutation
    risk_free_rate=0.0,
    seed=42,                  # reproducible: same seed, same p-value
    store_null_paths=100,     # permuted equity paths kept for the fan chart
)

result = MCPTRunner(config).run_portfolio(strategy.portfolio_data)

print(result.summary())
result.save(f"reports/{strategy.strategy_name}/validation")

# Pure form, for weights you built yourself:
result = MCPTRunner(config).run(weights_df, instrument_returns_df,
                                periods_per_year=252)

And it answers with a self-contained verdict block (illustrative run):

════════════════════════════════════════════════════════════════
Monte Carlo Permutation Test (MCPT)
════════════════════════════════════════════════════════════════
Mode:               signal_timing
Weight null:        circular_shift
Metric:             sharpe
Permutations:       1000 (1000 finite)
Seed:               42
────────────────────────────────────────────────────────────────
Observed sharpe:    0.1348  (annualized: 2.14)
Null distribution:  median -0.0032, 5–95% [-0.0501, 0.0489], 99% 0.0714
z-score vs null:    5.53
p-value:            0.0033
Verdict:            strong evidence of calendar-alignment skill (p < 0.01)
────────────────────────────────────────────────────────────────
NOTE: MCPT is an in-sample significance test. Combine with
walk-forward OOS, Deflated Sharpe and rank stability before trusting it.
════════════════════════════════════════════════════════════════

How to read it. The null's median per-day Sharpe is −0.0032 — mildly negative, which already teaches something: with these particular decisions, alternative calendar alignments slightly lose on this market, so beating zero was never the right bar. The luckiest 1% of shifts reached 0.0714. The real strategy sits at 0.1348 — nearly twice that percentile, 5.5 null standard deviations above the null mean, matched by 2 or 3 shifts in a thousand. That is evidence against the pre-committed circular-shift null. The two saved charts say the same thing visually: a histogram with the strategy far in the red-shaded tail, and an equity fan where the real curve climbs out of the grey band of shifted clones and never comes back.

Test 2 — shuffle the market itself

Test 1 shuffled the decisions and kept the market. The second test does the opposite: keep the strategy rule, rebuild the market 500 times, and recompute the rule from scratch on every rebuilt market — indicators, signals, weights, everything.

If the rule earns as much on structureless markets as on the real one, its "edge" never came from reading the market — it came from construction: being long often, being diversified, being levered. If the edge dies on the shuffled markets, it needed real temporal structure to exist. This tests a different null hypothesis from Test 1; it is not mathematically “strictly stronger”.

def weight_fn(prices: pd.DataFrame) -> pd.DataFrame:
    """The SAME strategy as a pure function of the prices it receives."""
    fast = prices.rolling(50).mean()
    slow = prices.rolling(200).mean()
    active = (fast > slow) & fast.notna() & slow.notna()
    counts = active.sum(axis=1)
    return active.astype(float).div(counts, axis=0).fillna(0.0)   # warm-up = 0.0, not NaN

config = MCPTConfig(mode="bar_permutation", n_permutations=500, seed=42)
result = MCPTRunner(config).run_bars(prices, weight_fn)

The one hard rule is purity: weight_fn is called once per rebuilt market and must compute everything — indicators, thresholds, normalizations, volatility scales — from the frame it receives. Anything precomputed outside the function was computed on the real market, and smuggles it into the null. The engine rejects what it can detect (non-finite weights raise immediately); referential purity is the author's contract. Three real ways people break it:

# WRONG - three ways to silently invalidate the bar-permutation null

sma_precomputed = prices.rolling(50).mean()          # (1) computed on the REAL market
def weight_fn_bad(p):
    return (sma_precomputed > p.rolling(200).mean()).astype(float)

vol_target = prices.pct_change().std()               # (2) full-sample constant
def weight_fn_bad2(p):
    return signals(p).div(vol_target, axis=1)        #     smuggles real-market scale

def weight_fn_bad3(p):
    w = signals(p)
    return w.where(w > 0)                            # (3) NaN warm-up rows -> ValueError
                                                     #     (fill with 0.0 instead)

Test 3 — maybe the search got lucky, not the strategy

One honesty problem remains, and it is the biggest one in practice. You did not write one strategy. You tried twenty variants and kept the best. Tests 1 and 2 validate the winner — but the winner was selected for looking good, so its p-value quietly inherits the luck of the whole search.

Feel the size of the effect: evaluate twenty worthless variants and pick the best Sharpe. That best-of-twenty is not centred at zero — the expected maximum of twenty noisy draws is substantially positive, always. Selection bias is not a flaw of the winning strategy; it is a property of the process that produced it. So the honest test must put the process itself on trial.

The size of the effect has a formula. If N worthless candidates have Sharpes that are pure noise of size σ, the expected value of the best one grows with the square root of the logarithm of N:

The square-root expression is only the leading asymptotic term. For finite N, the refined extreme-value formula used by the DSR calculation gives about 1.90σ for N=20 and 3.05σ for N=500. The expected maximum still rises with search size — which is why the search must sit inside the test — but the finite-sample numbers should come from the refined formula rather than the leading term.

Wrap the entire research process — the grid, the Optuna study, the selection rule — in one function that takes prices and returns the winner's score. Run it once on the real market, then on 200 shuffled markets, letting it search and select each time. The comparison becomes best-of-search versus best-of-search-on-noise — selection sits on both sides, and cancels:

PARAM_GRID = [(20, 100), (20, 150), (50, 150), (50, 200), (100, 200), (100, 250)]

def candidate_returns(prices: pd.DataFrame, fast: int, slow: int) -> pd.Series:
    f, s = prices.rolling(fast).mean(), prices.rolling(slow).mean()
    active = ((f > s) & f.notna() & s.notna()).astype(float)
    weights = active.div(active.sum(axis=1).where(lambda c: c > 0, 1.0), axis=0)
    return (weights.shift(1) * prices.pct_change()).sum(axis=1).fillna(0.0)

def train_fn(prices: pd.DataFrame) -> float:
    """The WHOLE research process: search the grid, return the winner's score."""
    best = -np.inf
    for fast, slow in PARAM_GRID:
        r = candidate_returns(prices, fast, slow).to_numpy()
        sd = r.std(ddof=1)
        if sd > 0:
            best = max(best, float(r.mean() / sd))
    return best

config = MCPTConfig(mode="training_process", n_permutations=200, seed=42)
result = MCPTRunner(config).run_training(prices, train_fn)

If train_fn's best-on-real-data does not clearly beat its best-on-noise, the pipeline manufactures its own alpha — and it will manufacture it again on live data, where the market supplies fresh noise daily. This test costs n_permutations × one full search. The observed market and every permuted market must use the identical search space, trial budget, pruning policy, stopping rule and winner-selection logic. To reduce compute, reduce the budget symmetrically on both sides or reduce the number of permutations.

When many candidates survive: the family problem

Test 3 judges the process as a whole. Often you want the finer answer: out of my twenty candidates, which ones individually are real? Answering that one-by-one at p < 0.05 is how "alpha farms" are born.

The arithmetic is brutal. Test one worthless strategy at the 5% level and you have a 5% chance of a false discovery. Test K of them and the chance that at least one slips through is:

The classical fix, Bonferroni, demands p < 0.05/K from every candidate. It works, but it treats the candidates as K independent bets — and they are not. SMA(50/200) and SMA(60/190) are nearly the same strategy; punishing them as two separate discoveries throws away real statistical power. The Romano–Wolf stepdown keeps the same family-wise protection while measuring how related the candidates actually are, instead of assuming the worst. The mechanism, in four steps:

1. One statistic per candidate

   θ̂_k = per-period Sharpe (or mean return) of candidate k, with H₀_k: θ_k ≤ 0.

2. Resample the family together

   Every bootstrap draw picks ONE shared index sequence (stationary blocks, expected length √T) applied to all K columns — correlated strategies rise and fall together in the resamples, exactly as they do in reality. Centered statistics δ_k^(b) = θ̂_k^(b) − θ̂_k form the null.

3. Rank against the family maximum, stepping down

   Sort candidates best-first. The best is ranked against max_j∈S δ_j^(b) over the full active set S; each processed candidate leaves S, shrinking the bar for the rest: p = (1 + #{b : max δ^(b) ≥ θ̂})/(B+1).

4. Force monotonicity

   Adjusted p-values never decrease down the ranking — a weaker candidate can never look more significant than a stronger one.

The two central objects in symbols. First the centered bootstrap statistic — candidate k's statistic in resample b, minus its observed value — which is what the candidate's pure noise looks like:

Then the stepdown p-value: candidate k is ranked not against its own noise but against the maximum noise across the still-active set S — the "did anyone in the family fluctuate this high?" question:

from backtester.validation import romano_wolf_stepdown

# T x K matrix: per-period returns, one column per candidate strategy
family = pd.DataFrame(
    {f"sma_{f}_{s}": candidate_returns(prices, f, s) for f, s in PARAM_GRID}
)

table = romano_wolf_stepdown(
    family,
    statistic="sharpe",     # per-period Sharpe, H0: true Sharpe <= 0
    n_resamples=1000,       # joint stationary block bootstrap
    seed=42,
)
print(table)
survivors = table[table.p_adjusted < 0.05]

              statistic  statistic_annualized  p_naive  p_adjusted  rank
sma_50_200       0.0781                1.2399   0.0040      0.0190     1
sma_20_150       0.0562                0.8925   0.0210      0.0829     2
sma_100_200      0.0488                0.7742   0.0340      0.0829     3
sma_20_100       0.0431                0.6842   0.0479      0.0999     4
sma_50_150       0.0402                0.6379   0.0619      0.1099     5
sma_100_250      0.0195                0.3098   0.2098      0.3187     6

# illustrative output - three candidates clear the naive 5% bar,
# ONE survives the family-wise correction

The two p-value columns are the whole story. p_naive is what you would report if each candidate were the only strategy you had ever tested — three of them clear 5%. p_adjusted is what survives the family — exactly one. The gap between the columns is your selection bias, measured. Deploy from the right column only. And because correlated near-duplicates move together in the joint resamples, they count as one discovery — two identical columns receive identical adjusted p-values, where Bonferroni would charge them twice.

PSR, DSR and the difference between CSCV PBO and rolling rank stability

These diagnostics ask related but different questions. The standalone functions implement PSR and the classical DSR formula for a supplied common trial population. Walk-forward reports PSR at N=1 or an explicitly labelled pooled DSR-style extension, plus rolling top-K rank stability. Canonical CSCV PBO is explained here for comparison; the current walk-forward engine does not claim to compute it.

PSR — how sure are we the true Sharpe is positive?

The Probabilistic Sharpe Ratio converts an estimated Sharpe into a probability, charging it for short samples and for non-normal returns. SR̂ is the per-day estimate, T the number of days behind it, γ₃ the skewness, γ₄ the raw kurtosis of the returns (3 = normal), Φ the standard normal CDF:

Fat tails (γ₄ > 3) and negative skew inflate the denominator and pull the probability down — the returns pay for their own non-normality. One unit trap: SR̂ must be per-period (daily) with T in days; feeding an annualized Sharpe with daily T massively overstates confidence.

DSR — the same, after admitting how many things you tried

If an optimizer evaluated N effectively independent trials and you report the best, the fair benchmark is no longer zero — it is the Sharpe that the luckiest of N worthless trials would show. That expected maximum has a closed form (γ ≈ 0.5772 is the Euler–Mascheroni constant, V[SR] the variance of Sharpe across the trials):

The Deflated Sharpe Ratio is the PSR measured against that raised bar. In the cited construction, the estimated trial Sharpes belong to one common strategy population and N is the effective number of independent trials, not automatically the raw count of correlated Optuna variants. The standalone helper implements that formula for a supplied common trial population. The walk-forward engine instead pools finite objectives from multiple chronological training folds; this is a useful extension, but not the canonical single-population construction, so 0.12.1 labels it pooled_walk_forward_dsr_style. It accepts dsr_effective_n_trials; if dependence is not estimated it reports and uses the raw finite completed count as an approximation. Failed and pruned trials remain part of the research audit, but they cannot estimate the cross-trial Sharpe variance without a finite objective. With effective N = 1, the engine labels the result probabilistic_sharpe_n1 because the multiple-testing deflation has vanished.

from backtester.walkforward.statistics import deflated_sharpe, probabilistic_sharpe

# PSR: probability the TRUE Sharpe exceeds a benchmark, given T, skew, kurtosis
psr = probabilistic_sharpe(
    observed_sr=0.10,        # per-period (daily) Sharpe - NOT annualized
    benchmark_sr=0.0,
    n_obs=2520,              # T behind the estimate
    skewness=-0.8,           # fat left tail lowers the probability
    kurtosis=6.0,            # raw kurtosis (3 = normal)
)

# DSR: PSR against the expected MAX Sharpe of N trials under the null
dsr = deflated_sharpe(
    trial_sharpes=completed_trial_sharpes,  # finite completed objectives estimate V[SR]
    n_trials=len(completed_trial_sharpes),  # raw completed-trial approximation
    effective_n_trials=effective_n,         # optional; omit for conservative raw N
    observed_sr=best_daily_sharpe,
    n_obs=2520,
    skewness=skew, kurtosis=kurt_raw,
)
# DSR >= 0.95 robust | 0.80-0.95 marginal | < 0.80 likely false positive

Canonical CSCV PBO — does the in-sample winner keep winning?

The Probability of Backtest Overfitting asks a rank question across symmetric train/test combinations built from the same complete T × N performance matrix. For every CSCV combination, select the winner in-sample and rank that same configuration out-of-sample among all N configurations. If ω is its relative OOS rank (1 = still the best, 0 = the worst) and λ = ln(ω/(1−ω)) its logit:

Concrete values make the logit tangible — rank 90% maps to +2.2, the coin-flip rank 50% to exactly zero, rank 20% deep below:

Canonical PBO is the fraction of CSCV logits at or below zero. It requires the full configuration matrix and symmetric complementary splits; ordinary rolling folds plus a preselected top-K subset are a different estimator.

The engine metric — rolling top-K OOS rank failure

The engine re-runs the top K in-sample trials on each rolling fold, ranks the IS winner only within that subset, and reports the fraction of finite fold logits at or below zero as walk_forward_top_k_rank_failure_rate. This is a useful rank-stability sensitivity check, but it is not canonical CSCV PBO and canonical PBO thresholds must not be attached to it. The old pbo_trials and result.pbo names remain deprecated compatibility aliases during 0.12.x. When candidate OOS evaluations are unavailable, the metric returns None with a reason.

The pooled DSR-style and rank-stability diagnostics wire into walk-forward with two config flags; the bootstrap CI comes along for free:

from backtester.walkforward import WalkForwardConfig, WalkForwardEngine

# The walk-forward computes an explicitly labelled pooled DSR-style
# diagnostic and rolling top-K rank stability. Neither is canonical CSCV PBO.
config = WalkForwardConfig(
    scheme="rolling",
    train_months=24, test_months=6, step_months=6,
    purge_days=5, extra_pre_oos_purge_pct=0.01,
    compute_deflated_sharpe=True,
    dsr_effective_n_trials=None,  # raw completed count when dependence is unknown
    compute_rank_stability=True,
    rank_stability_trials=8,
    seed=42,
)

engine = WalkForwardEngine(config=config,
                           backtester_factory=factory,   # per-fold refit
                           optimizer=optimizer)
result = engine.run(strategy.portfolio_data)

print(result.summary())
result.deflated_sharpe    # probability in [0, 1]
result.deflated_sharpe_method
# "pooled_walk_forward_dsr_style" (or "probabilistic_sharpe_n1")
result.walk_forward_top_k_rank_failure_rate
result.rank_stability_reason     # explains None when unavailable
result.sharpe_ci_5pct, result.sharpe_ci_95pct   # bootstrap CI, built in

Fold geometry, pre-OOS purging, and the honest-reporting rules behind these numbers live in the walk-forward case study; how the Optuna trial population feeds the DSR is in the Optuna guide.

Determinism and the audit trail

A p-value you cannot reproduce is an anecdote with decimals. Every method on this page is built to replay exactly.

Three mechanisms make that hold. Seeds are threaded, never hardcoded: every permutation, bootstrap resample and optimizer draw consumes the run's explicit seed, so the same configuration produces bit-identical results on any machine. Every artifact is fingerprinted: the run's config and outcome hash into a SHA-256 fingerprint stored in mcpt.json — two reports with the same fingerprint are the same experiment, by construction. Honest edge cases: the +1 in the p-value formula means no test can claim certainty; NaN permutations are excluded and counted in warnings rather than silently absorbed; rolling rank stability says "unavailable" instead of inventing a zero.

# mcpt.json - the audit artifact every run writes
{
  "mode": "signal_timing",
  "metric": "sharpe",
  "observed": 0.1348,
  "observed_annualized": 2.1400,
  "p_value": 0.0033,
  "z_score": 5.53,
  "n_permutations": 1000,
  "null_quantiles": {"q01": -0.071, "q50": -0.003, "q99": 0.071},
  "seed": 42,
  "verdict": "strong evidence of calendar-alignment skill (p < 0.01)",
  "config": {"mode": "signal_timing", "weight_null": "circular_shift",
             "n_permutations": 1000, "seed": 42},
  "fingerprint": "9644ac9918ab05cb"
}

# Same seed -> same p-value -> same fingerprint. Which makes the test
# a CI gate:
def test_strategy_still_beats_its_null():
    result = MCPTRunner(MCPTConfig(n_permutations=500, seed=42)).run(W, R)
    assert result.p_value < 0.05, result.verdict

The CI-gate pattern is worth adopting early: pin the seed, assert the p-value, and any code change that quietly destroys the strategy's edge — a refactored indicator, a shifted rebalance date, a data revision — fails the build instead of shipping to production with a beautiful, meaningless equity curve.

Reading the outputs, and four traps

Every MCPT run saves the machine-readable verdict, the full null distribution as CSV, and two charts: the histogram (your strategy against the blue mass of its circular-shift null paths, p-value tail shaded red) and the equity fan (the real curve against the grey band of clone equity curves).

# Three runnable examples ship with the engine
./strategy.sh example_mcpt_01_signal_timing      # circular-shift decisions vs returns
./strategy.sh example_mcpt_02_bar_permutation    # rebuild the strategy on permuted markets
./strategy.sh example_mcpt_03_training_process   # permute the whole grid search + Romano-Wolf

# Environment knobs
QJ_MCPT_N=200 ./strategy.sh example_mcpt_01_signal_timing      # fewer permutations
QJ_MCPT_COST_BPS=10 ./strategy.sh example_mcpt_01_signal_timing  # net of 10 bps on turnover

The deployment checklist

Gates 1–4 are in-sample significance (this guide); gates 5–7 are out-of-sample persistence (the walk-forward guide). A strategy needs both legs, and every gate is cheap compared to trading a false positive.

1. Signal-timing MCPT p < 0.05

   The decision timing beats its circular-shift null paths on the real market; report block and full-row sensitivity separately.

2. Bar-permutation MCPT p < 0.05

   The edge dies on structureless synthetic markets — it needed real temporal structure to exist.

3. Training-process MCPT p < 0.05

   When a search was involved: the whole process beats the same process run on noise.

4. Romano–Wolf p_adjusted < 0.05

   The chosen candidate survives its own family — not just the naive single-test p.

5. Walk-forward OOS holds up

   OOS Sharpe > 0, overfit ratio < 2.5 and efficiency > 0.4 across date-bounded folds with an explicit pre-OOS purge.

6. Sharpe selection method is disclosed

   Use canonical DSR only for one documented trial population; label pooled walk-forward calculations DSR-style and archive both the raw completed count and effective-N assumption.

7. Rank stability is disclosed

   Report the rolling top-K failure rate as a sensitivity diagnostic, or compute canonical CSCV PBO from a complete trial matrix; do not mix their names or thresholds.

References

• Bailey, D. H. & López de Prado, M. (2012). The Sharpe Ratio Efficient Frontier. Journal of Risk 15(2) — the Probabilistic Sharpe Ratio.
• Bailey, D. H. & López de Prado, M. (2014). The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality. Journal of Portfolio Management 40(5).
• Bailey, D. H., Borwein, J., López de Prado, M. & Zhu, Q. J. (2017). The Probability of Backtest Overfitting. Journal of Computational Finance 20(4).
• Lo, A. W. (2002). The Statistics of Sharpe Ratios. Financial Analysts Journal 58(4).
• Masters, T. (2018). Permutation and Randomization Tests for Trading System Development.
• Politis, D. N. & Romano, J. P. (1994). The Stationary Bootstrap. Journal of the American Statistical Association 89(428).
• Romano, J. P. & Wolf, M. (2005). Stepwise Multiple Testing as Formalized Data Snooping. Econometrica 73(4); and (2016) Efficient computation of adjusted p-values for resampling-based stepdown multiple testing. Statistics & Probability Letters 113.

Continue the research