HKUDS/Vibe-Trading · error · ValueError

n_splits must be an even number >= 4, got {n_splits}

Error message

n_splits must be an even number >= 4, got {n_splits}

What it means

probability_of_backtest_overfitting implements CSCV, which needs the return series split into an even number n_splits >= 4 of blocks so they can be recombined into symmetric in-sample/out-of-sample halves. An odd or too-small count would break the combinatorial pairing logic and bias the PBO estimate, so it is rejected up front.

Source

Thrown at agent/src/quantlib/multipletesting.py:478

        performance: Returns per strategy, rows as observations and columns as
            strategies. A DataFrame's column labels are preserved only for the
            caller's convenience; this function returns no per-strategy output.
        n_splits: Number of subsets. Must be even and at least 4; the number of
            combinations is ``C(n_splits, n_splits/2)``, so 16 gives 12,870.
        ddof: Delta degrees of freedom for the Sharpe standard deviation,
            forwarded to :func:`sharpe_ratio`.

    Returns:
        A :class:`CSCVResult`.

    Raises:
        ValueError: If ``n_splits`` is odd or below 4, if fewer than 2
            strategies are supplied (a rank needs competitors), if the sample
            cannot give each subset at least 2 rows, or if every strategy has
            zero variance so no Sharpe is defined.
    """
    if n_splits < 4 or n_splits % 2 != 0:
        raise ValueError(f"n_splits must be an even number >= 4, got {n_splits}")

    frame = pd.DataFrame(performance)
    matrix = frame.to_numpy(dtype=float)
    if matrix.ndim != 2:
        raise ValueError(f"performance must be 2-D, got shape {matrix.shape}")

    n_rows, n_strategies = matrix.shape
    if n_strategies < 2:
        raise ValueError(
            f"CSCV ranks strategies against each other and needs at least 2, "
            f"got {n_strategies}"
        )

    subset_size = n_rows // n_splits
    if subset_size < 2:
        raise ValueError(
            f"{n_rows} rows split {n_splits} ways gives {subset_size} row(s) per "
            "subset; each subset needs at least 2 for a Sharpe"

View on GitHub (pinned to 80ffdda44c)

Solutions

1. Use an even value >= 4, e.g. n_splits=16 as in Bailey et al.'s CSCV paper.
2. Validate/round the config value at load time: n_splits = max(4, 2 * (n_splits // 2)).
3. Document the even-and->=4 constraint next to any user-facing parameter.

Example fix

# before
result = probability_of_backtest_overfitting(perf, n_splits=5)

# after
result = probability_of_backtest_overfitting(perf, n_splits=16)

n_splits = int(n_splits)
if n_splits < 4 or n_splits % 2:
    n_splits = max(4, 2 * (n_splits // 2))  # snap to nearest valid even >= 4

def valid_cscv_splits(n: int) -> bool:
    return isinstance(n, int) and n >= 4 and n % 2 == 0

try:
    pbo = probability_of_backtest_overfitting(perf, n_splits)
except ValueError as e:
    if 'n_splits' in str(e):
        pbo = probability_of_backtest_overfitting(perf, n_splits=16)  # safe default
    else:
        raise

When it happens

Trigger: Calling probability_of_backtest_overfitting(performance, n_splits=3) or n_splits=5 (odd), or n_splits=2 (below the minimum of 4).

Common situations: Copying n_splits from a K-fold cross-validation config (often 3, 5, or 10) where odd values are the norm; exposing n_splits as a user-tunable knob in a backtest UI without documenting the parity constraint.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• CSCV ranks strategies against each other and needs at least
• performance must be 2-D, got shape {matrix.shape}
• {n_rows} rows split {n_splits} ways gives {subset_size} row(
• no split produced a usable Sharpe; every strategy may have z
• delay requires n >= 1 (lookahead ban)

AI-assisted analysis of HKUDS/Vibe-Trading@80ffdda44c (2026-08-28). Data as JSON: /api/errors/5e49dba96d782d59. Report an issue: GitHub.