{"record":{"id":"b598983ef0cc4263","repo":"keras-team/keras","slug":"if-pad-to-max-tokens-is-true-must-set-max-tokens","errorCode":null,"errorMessage":"If pad_to_max_tokens is True, must set `max_tokens`. Received: max_tokens={max_tokens}","messagePattern":"If pad_to_max_tokens is True, must set `max_tokens`\\. Received: max_tokens=(.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"keras/src/layers/preprocessing/index_lookup.py","lineNumber":139,"sourceCode":"        invert=False,\n        output_mode=\"int\",\n        sparse=False,\n        pad_to_max_tokens=False,\n        oov_method=\"floormod\",\n        name=None,\n        salt=None,\n        **kwargs,\n    ):\n        # If max_tokens is set, the value must be greater than 1 - otherwise we\n        # are creating a 0-element vocab, which doesn't make sense.\n        if max_tokens is not None and max_tokens <= 1:\n            raise ValueError(\n                \"If set, `max_tokens` must be greater than 1. \"\n                f\"Received: max_tokens={max_tokens}\"\n            )\n\n        if pad_to_max_tokens and max_tokens is None:\n            raise ValueError(\n                \"If pad_to_max_tokens is True, must set `max_tokens`. \"\n                f\"Received: max_tokens={max_tokens}\"\n            )\n\n        if num_oov_indices < 0:\n            raise ValueError(\n                \"`num_oov_indices` must be greater than or equal to 0. \"\n                f\"Received: num_oov_indices={num_oov_indices}\"\n            )\n\n        argument_validation.validate_string_arg(\n            oov_method,\n            allowable_strings=(\"floormod\", \"farmhash\"),\n            caller_name=self.__class__.__name__,\n            arg_name=\"oov_method\",\n        )\n\n        if salt is not None:","sourceCodeStart":121,"sourceCodeEnd":157,"githubUrl":"https://github.com/keras-team/keras/blob/7a34a03db60bf60042242d6a556fc3be119046a5/keras/src/layers/preprocessing/index_lookup.py#L121-L157","documentation":"Raised by IndexLookup's constructor when `pad_to_max_tokens=True` but `max_tokens` is None. Padding the output to a fixed width is only defined if that width (max_tokens) is known at construction time.","triggerScenarios":"Calling StringLookup(vocabulary=vocab, pad_to_max_tokens=True) without setting max_tokens; same for IntegerLookup and CategoryEncoding.","commonSituations":"Enabling pad_to_max_tokens for batch-shape stability (e.g. before a Transformer encoder) while forgetting the vocabulary size is unknown with an adaptive vocabulary.","solutions":["Set an explicit max_tokens > 1, e.g. StringLookup(vocabulary=vocab, max_tokens=20000, pad_to_max_tokens=True)","If fixed-width output isn't required, drop pad_to_max_tokens (it defaults to False)"],"exampleFix":"// before\nlayer = StringLookup(vocabulary=vocab, pad_to_max_tokens=True)\n// after\nlayer = StringLookup(vocabulary=vocab, max_tokens=20000, pad_to_max_tokens=True)","handlingStrategy":"validation","validationCode":"if pad_to_max_tokens:\n    assert max_tokens is not None and max_tokens > 1, 'pad_to_max_tokens requires max_tokens'","typeGuard":"def valid_lookup_config(pad, mt):\n    return not pad or (isinstance(mt, int) and mt > 1)","tryCatchPattern":"try:\n    layer = StringLookup(vocabulary=vocab, max_tokens=mt, pad_to_max_tokens=True)\nexcept ValueError:\n    layer = StringLookup(vocabulary=vocab, max_tokens=len(vocab) + 2, pad_to_max_tokens=True)","preventionTips":["Set an explicit max_tokens whenever fixed-width dense output is needed (e.g. before Transformers)","Compute max_tokens as len(vocab) + num_oov_indices + 1 when adapting"],"tags":["keras","preprocessing","index-lookup","validation"],"backgroundTag":"missing-required-argument","analyzedSha":"7a34a03db60bf60042242d6a556fc3be119046a5","analyzedAt":"2026-08-25T21:25:25.994Z","schemaVersion":2},"datasetVersion":"2026-08-26T02:17:13.382Z"}