rohitg00/ai-engineering-from-scratch · error · ValueError

correct must be 0/1 or boolean

Error message

correct must be 0/1 or boolean

What it means

Calibration correctness labels must be binary (0/1 or boolean) because ECE and Brier compare each label against the binned confidence as a frequency; continuous targets like accuracy values or multi-class ids would corrupt every bin statistic, so _validate_probs checks the unique values.

Solutions

1. Convert labels: correct = int(pred == gold)
2. For multi-class, mark correctness per top-1 prediction before calling the metric
3. Round continuous targets to 0/1 only if they genuinely represent soft outcomes, otherwise use a proper regression calibration metric

When it happens

Trigger: Thrown at phases/19-capstone-projects/73-perplexity-calibration/code/main.py:68 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of rohitg00/ai-engineering-from-scratch@39ea8a1c6d (2026-08-26). Data as JSON: /api/errors/5aeb3fe0bdd56d54. Report an issue: GitHub.

Appendix: source

Thrown at phases/19-capstone-projects/73-perplexity-calibration/code/main.py:68

        return cls(perplexity=math.exp(avg_nll), avg_neg_log_likelihood=avg_nll, total_tokens=total_tokens)


def perplexity(neg_log_probs: Sequence[float], token_counts: Sequence[int]) -> float:
    return PerplexityResult.from_token_nll(neg_log_probs, token_counts).perplexity


def _validate_probs(confidences: np.ndarray, correct: np.ndarray) -> None:
    if confidences.shape != correct.shape:
        raise ValueError("confidences and correct must have the same shape")
    if confidences.ndim != 1:
        raise ValueError("confidences must be 1-D")
    if confidences.size == 0:
        return
    if float(confidences.min()) < 0.0 or float(confidences.max()) > 1.0:
        raise ValueError("confidences must lie in [0, 1]")
    uniq = set(np.unique(correct).tolist())
    if not uniq.issubset({0, 1, 0.0, 1.0, True, False}):
        raise ValueError("correct must be 0/1 or boolean")


def _bin_indices(confidences: np.ndarray, n_bins: int) -> np.ndarray:
    edges = np.linspace(0.0, 1.0, n_bins + 1)
    idx = np.searchsorted(edges, confidences, side="right") - 1
    idx = np.clip(idx, 0, n_bins - 1)
    return idx


def expected_calibration_error(confidences: Sequence[float], correct: Sequence[int], bins: int = 10) -> tuple[float, int]:
    conf = np.asarray(confidences, dtype=np.float64)
    corr = np.asarray(correct, dtype=np.float64)
    _validate_probs(conf, corr)
    if bins <= 0:
        raise ValueError("bins must be positive")
    n = conf.size
    if n == 0:
        return (0.0, 0)

View on GitHub (pinned to 39ea8a1c6d)