{"record":{"id":"017241da0753513f","repo":"TheAlgorithms/Python","slug":"label-value-must-not-be-greater-than-vocabulary-si","errorCode":null,"errorMessage":"Label value must not be greater than vocabulary size.","messagePattern":"Label value must not be greater than vocabulary size\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"machine_learning/loss_functions.py","lineNumber":560,"sourceCode":"    ...    [[[0.28, 0.19, 0.21 , 0.15, 0.15],\n    ...      [0.24, 0.19, 0.09, 0.18, 0.27]],\n    ...      [[0.03, 0.26, 0.21, 0.18, 0.30],\n    ...       [0.28, 0.10, 0.33, 0.15, 0.12]]]\n    ... )\n    >>> perplexity_loss(y_true, y_pred)\n    Traceback (most recent call last):\n    ...\n    ValueError: Batch size of y_true and y_pred must be equal.\n    \"\"\"\n\n    vocab_size = y_pred.shape[2]\n\n    if y_true.shape[0] != y_pred.shape[0]:\n        raise ValueError(\"Batch size of y_true and y_pred must be equal.\")\n    if y_true.shape[1] != y_pred.shape[1]:\n        raise ValueError(\"Sentence length of y_true and y_pred must be equal.\")\n    if np.max(y_true) > vocab_size:\n        raise ValueError(\"Label value must not be greater than vocabulary size.\")\n\n    # Matrix to select prediction value only for true class\n    filter_matrix = np.array(\n        [[list(np.eye(vocab_size)[word]) for word in sentence] for sentence in y_true]\n    )\n\n    # Getting the matrix containing prediction for only true class\n    true_class_pred = np.sum(y_pred * filter_matrix, axis=2).clip(epsilon, 1)\n\n    # Calculating perplexity for each sentence\n    perp_losses = np.exp(np.negative(np.mean(np.log(true_class_pred), axis=1)))\n\n    return np.mean(perp_losses)\n\n\ndef smooth_l1_loss(y_true: np.ndarray, y_pred: np.ndarray, beta: float = 1.0) -> float:\n    \"\"\"\n    Calculate the Smooth L1 Loss between y_true and y_pred.","sourceCodeStart":542,"sourceCodeEnd":578,"githubUrl":"https://github.com/TheAlgorithms/Python/blob/f5988cc09713315817df6a7e327e258013a94440/machine_learning/loss_functions.py#L542-L578","documentation":"Thrown by perplexity_loss when any token id in y_true exceeds vocab_size (y_pred.shape[2]). Token ids index rows of np.eye(vocab_size) to select the predicted probability of the true class, so an id >= vocab_size would raise IndexError downstream; the function guards this explicitly.","triggerScenarios":"Calling perplexity_loss with y_true containing an id such as 500 while y_pred's last dimension (vocabulary size) is 300; ids built with a larger vocabulary than the prediction matrix.","commonSituations":"Vocabulary rebuilt/resized after tokenization (model trained on fewer tokens); tokenizer trained on train split but evaluation text contains rare tokens mapped to high ids; off-by-one where vocab includes id == vocab_size.","solutions":["Make vocab_size consistent: y_pred must have shape (..., len(vocab)) matching the tokenizer used for y_true.","Re-tokenize y_true with the same vocabulary that produced the model outputs.","Map or drop out-of-vocabulary ids: y_true = np.where(y_true < vocab_size, y_true, unk_id)."],"exampleFix":"# before\nvocab_size = 300\ny_true = np.array([[0, 250, 499]])   # 499 out of range\nloss = perplexity_loss(y_true, y_pred)\n\n# after\ny_true = np.where(y_true < vocab_size, y_true, 1)  # map to <unk>\nloss = perplexity_loss(y_true, y_pred)","handlingStrategy":"validation","validationCode":"vocab_size = y_pred.shape[2]\nassert y_true.max() < vocab_size, f\"label {y_true.max()} >= vocab {vocab_size}\"\nloss = perplexity_loss(y_true, y_pred)","typeGuard":"def labels_in_vocab(y_true: np.ndarray, vocab_size: int) -> bool:\n    return int(y_true.max()) < vocab_size and int(y_true.min()) >= 0","tryCatchPattern":"try:\n    loss = perplexity_loss(y_true, y_pred)\nexcept ValueError as e:\n    if \"vocabulary size\" in str(e):\n        unk = 1  # or any valid <unk> id\n        y_true = np.where(y_true < y_pred.shape[2], y_true, unk)\n        loss = perplexity_loss(y_true, y_pred)\n    else:\n        raise","preventionTips":["Tokenize references with the exact tokenizer/vocabulary used to build the model.","Persist the vocabulary with checkpoints and reload both together.","Map out-of-vocabulary ids to a dedicated <unk> token during decoding."],"tags":["machine-learning","loss-function","nlp","vocabulary"],"backgroundTag":null,"analyzedSha":"f5988cc09713315817df6a7e327e258013a94440","analyzedAt":"2026-08-14T17:30:07.041Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}