TheAlgorithms/Python · error · ValueError

Label value must not be greater than vocabulary size.

Error message

Label value must not be greater than vocabulary size.

What it means

Thrown by perplexity_loss when any token id in y_true exceeds vocab_size (y_pred.shape[2]). Token ids index rows of np.eye(vocab_size) to select the predicted probability of the true class, so an id >= vocab_size would raise IndexError downstream; the function guards this explicitly.

Source

Thrown at machine_learning/loss_functions.py:560

    ...    [[[0.28, 0.19, 0.21 , 0.15, 0.15],
    ...      [0.24, 0.19, 0.09, 0.18, 0.27]],
    ...      [[0.03, 0.26, 0.21, 0.18, 0.30],
    ...       [0.28, 0.10, 0.33, 0.15, 0.12]]]
    ... )
    >>> perplexity_loss(y_true, y_pred)
    Traceback (most recent call last):
    ...
    ValueError: Batch size of y_true and y_pred must be equal.
    """

    vocab_size = y_pred.shape[2]

    if y_true.shape[0] != y_pred.shape[0]:
        raise ValueError("Batch size of y_true and y_pred must be equal.")
    if y_true.shape[1] != y_pred.shape[1]:
        raise ValueError("Sentence length of y_true and y_pred must be equal.")
    if np.max(y_true) > vocab_size:
        raise ValueError("Label value must not be greater than vocabulary size.")

    # Matrix to select prediction value only for true class
    filter_matrix = np.array(
        [[list(np.eye(vocab_size)[word]) for word in sentence] for sentence in y_true]
    )

    # Getting the matrix containing prediction for only true class
    true_class_pred = np.sum(y_pred * filter_matrix, axis=2).clip(epsilon, 1)

    # Calculating perplexity for each sentence
    perp_losses = np.exp(np.negative(np.mean(np.log(true_class_pred), axis=1)))

    return np.mean(perp_losses)


def smooth_l1_loss(y_true: np.ndarray, y_pred: np.ndarray, beta: float = 1.0) -> float:
    """
    Calculate the Smooth L1 Loss between y_true and y_pred.

View on GitHub (pinned to f5988cc097)

Solutions

1. Make vocab_size consistent: y_pred must have shape (..., len(vocab)) matching the tokenizer used for y_true.
2. Re-tokenize y_true with the same vocabulary that produced the model outputs.
3. Map or drop out-of-vocabulary ids: y_true = np.where(y_true < vocab_size, y_true, unk_id).

Example fix

# before
vocab_size = 300
y_true = np.array([[0, 250, 499]])   # 499 out of range
loss = perplexity_loss(y_true, y_pred)

# after
y_true = np.where(y_true < vocab_size, y_true, 1)  # map to <unk>
loss = perplexity_loss(y_true, y_pred)

vocab_size = y_pred.shape[2]
assert y_true.max() < vocab_size, f"label {y_true.max()} >= vocab {vocab_size}"
loss = perplexity_loss(y_true, y_pred)

def labels_in_vocab(y_true: np.ndarray, vocab_size: int) -> bool:
    return int(y_true.max()) < vocab_size and int(y_true.min()) >= 0

try:
    loss = perplexity_loss(y_true, y_pred)
except ValueError as e:
    if "vocabulary size" in str(e):
        unk = 1  # or any valid <unk> id
        y_true = np.where(y_true < y_pred.shape[2], y_true, unk)
        loss = perplexity_loss(y_true, y_pred)
    else:
        raise

When it happens

Trigger: Calling perplexity_loss with y_true containing an id such as 500 while y_pred's last dimension (vocabulary size) is 300; ids built with a larger vocabulary than the prediction matrix.

Common situations: Vocabulary rebuilt/resized after tokenization (model trained on fewer tokens); tokenizer trained on train split but evaluation text contains rare tokens mapped to high ids; off-by-one where vocab includes id == vocab_size.

Related errors

• Batch size of y_true and y_pred must be equal.
• Sentence length of y_true and y_pred must be equal.
• Input arrays must have the same length.
• Input arrays must have the same shape.
• y_true must be one-hot encoded.

AI-assisted analysis of TheAlgorithms/Python@f5988cc097 (2026-08-14). Data as JSON: /api/errors/017241da0753513f. Report an issue: GitHub.