rohitg00/ai-engineering-from-scratch · error · ValueError

paired arrays must align

Error message

paired arrays must align

What it means

bootstrap_pairwise_diff resamples paired per-task score differences between two models; unequal list lengths mean the arrays are not truly paired (missing tasks on one side), so element-wise differences would compare scores from different tasks and produce a garbage confidence interval.

Solutions

1. Join model A and B runs on task_id first, then pass the aligned arrays
2. Drop tasks scored by only one model before bootstrapping
3. Assert equal lengths in your aggregation pipeline before calling pairwise functions

When it happens

Trigger: Thrown at phases/19-capstone-projects/74-leaderboard-aggregation/code/main.py:128 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of rohitg00/ai-engineering-from-scratch@39ea8a1c6d (2026-08-26). Data as JSON: /api/errors/e856bb87fa3db25a. Report an issue: GitHub.

Appendix: source

Thrown at phases/19-capstone-projects/74-leaderboard-aggregation/code/main.py:128

    n = arr.size
    means = np.empty(b, dtype=np.float64)
    for i in range(b):
        idx = rng.integers(0, n, size=n)
        means[i] = float(arr[idx].mean())
    lo = float(np.percentile(means, 100.0 * (alpha / 2.0)))
    hi = float(np.percentile(means, 100.0 * (1.0 - alpha / 2.0)))
    return (lo, hi)


def bootstrap_pairwise_diff(
    paired_a: Sequence[float],
    paired_b: Sequence[float],
    b: int = 1000,
    alpha: float = 0.05,
    seed: int = 0,
) -> tuple[float, float, float]:
    if len(paired_a) != len(paired_b):
        raise ValueError("paired arrays must align")
    if len(paired_a) == 0:
        return (0.0, 0.0, 0.0)
    rng = np.random.default_rng(seed)
    a = np.asarray(paired_a, dtype=np.float64)
    b_arr = np.asarray(paired_b, dtype=np.float64)
    diffs = a - b_arr
    diff_mean = float(diffs.mean())
    n = diffs.size
    boots = np.empty(b, dtype=np.float64)
    for i in range(b):
        idx = rng.integers(0, n, size=n)
        boots[i] = float(diffs[idx].mean())
    lo = float(np.percentile(boots, 100.0 * (alpha / 2.0)))
    hi = float(np.percentile(boots, 100.0 * (1.0 - alpha / 2.0)))
    return (diff_mean, lo, hi)


def _win_rate(model_id: str, runs_by_task: dict[str, list[EvalRun]]) -> tuple[float, int]:

View on GitHub (pinned to 39ea8a1c6d)