rohitg00/ai-engineering-from-scratch · error · ValueError

neg_log_probs and token_counts must align

Error message

neg_log_probs and token_counts must align

What it means

Error "neg_log_probs and token_counts must align" thrown in rohitg00/ai-engineering-from-scratch.

Source

Thrown at phases/19-capstone-projects/73-perplexity-calibration/code/main.py:37

@dataclass
class PerplexityResult:
    perplexity: float
    avg_neg_log_likelihood: float
    total_tokens: int

    def to_dict(self) -> dict:
        return {
            "perplexity": self.perplexity,
            "avg_neg_log_likelihood": self.avg_neg_log_likelihood,
            "total_tokens": self.total_tokens,
        }

    @classmethod
    def from_token_nll(cls, neg_log_probs: Sequence[float], token_counts: Sequence[int]) -> "PerplexityResult":
        if len(neg_log_probs) != len(token_counts):
            raise ValueError("neg_log_probs and token_counts must align")
        total_nll = 0.0
        total_tokens = 0
        for nll, n in zip(neg_log_probs, token_counts):
            if nll < 0:
                raise ValueError("neg_log_probs must be non-negative (did you forget the negation?)")
            if n < 0:
                raise ValueError("token_counts must be non-negative")
            total_nll += float(nll)
            total_tokens += int(n)
        if total_tokens == 0:
            return cls(perplexity=float("nan"), avg_neg_log_likelihood=0.0, total_tokens=0)
        avg_nll = total_nll / total_tokens
        return cls(perplexity=math.exp(avg_nll), avg_neg_log_likelihood=avg_nll, total_tokens=total_tokens)


def perplexity(neg_log_probs: Sequence[float], token_counts: Sequence[int]) -> float:
    return PerplexityResult.from_token_nll(neg_log_probs, token_counts).perplexity

View on GitHub (pinned to 39ea8a1c6d)

When it happens

Trigger: Thrown at phases/19-capstone-projects/73-perplexity-calibration/code/main.py:37 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of rohitg00/ai-engineering-from-scratch@39ea8a1c6d (2026-08-26). Data as JSON: /api/errors/fea73c3dd17e6cf4. Report an issue: GitHub.