keras-team/keras · error · ValueError

If pad_to_max_tokens is True, must set `max_tokens`. Receive

Error message

If pad_to_max_tokens is True, must set `max_tokens`. Received: max_tokens={max_tokens}

What it means

Raised by IndexLookup's constructor when `pad_to_max_tokens=True` but `max_tokens` is None. Padding the output to a fixed width is only defined if that width (max_tokens) is known at construction time.

Source

Thrown at keras/src/layers/preprocessing/index_lookup.py:139

        invert=False,
        output_mode="int",
        sparse=False,
        pad_to_max_tokens=False,
        oov_method="floormod",
        name=None,
        salt=None,
        **kwargs,
    ):
        # If max_tokens is set, the value must be greater than 1 - otherwise we
        # are creating a 0-element vocab, which doesn't make sense.
        if max_tokens is not None and max_tokens <= 1:
            raise ValueError(
                "If set, `max_tokens` must be greater than 1. "
                f"Received: max_tokens={max_tokens}"
            )

        if pad_to_max_tokens and max_tokens is None:
            raise ValueError(
                "If pad_to_max_tokens is True, must set `max_tokens`. "
                f"Received: max_tokens={max_tokens}"
            )

        if num_oov_indices < 0:
            raise ValueError(
                "`num_oov_indices` must be greater than or equal to 0. "
                f"Received: num_oov_indices={num_oov_indices}"
            )

        argument_validation.validate_string_arg(
            oov_method,
            allowable_strings=("floormod", "farmhash"),
            caller_name=self.__class__.__name__,
            arg_name="oov_method",
        )

        if salt is not None:

View on GitHub (pinned to 7a34a03db6)

Solutions

Set an explicit max_tokens > 1, e.g. StringLookup(vocabulary=vocab, max_tokens=20000, pad_to_max_tokens=True)
If fixed-width output isn't required, drop pad_to_max_tokens (it defaults to False)

Example fix

// before
layer = StringLookup(vocabulary=vocab, pad_to_max_tokens=True)
// after
layer = StringLookup(vocabulary=vocab, max_tokens=20000, pad_to_max_tokens=True)

Defensive patterns

Strategy: validation

Validate before calling

if pad_to_max_tokens:
    assert max_tokens is not None and max_tokens > 1, 'pad_to_max_tokens requires max_tokens'

Type guard

def valid_lookup_config(pad, mt):
    return not pad or (isinstance(mt, int) and mt > 1)

Try / catch

try:
    layer = StringLookup(vocabulary=vocab, max_tokens=mt, pad_to_max_tokens=True)
except ValueError:
    layer = StringLookup(vocabulary=vocab, max_tokens=len(vocab) + 2, pad_to_max_tokens=True)

Prevention

Set an explicit max_tokens whenever fixed-width dense output is needed (e.g. before Transformers)
Compute max_tokens as len(vocab) + num_oov_indices + 1 when adapting

When it happens

Trigger: Calling StringLookup(vocabulary=vocab, pad_to_max_tokens=True) without setting max_tokens; same for IntegerLookup and CategoryEncoding.

Common situations: Enabling pad_to_max_tokens for batch-shape stability (e.g. before a Transformer encoder) while forgetting the vocabulary size is unknown with an adaptive vocabulary.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

AI-assisted analysis of keras-team/keras@7a34a03db6 (2026-08-25). Data as JSON: /api/errors/b598983ef0cc4263. Report an issue: GitHub.