huggingface/transformers · error · NotImplementedError

prefix_allowed_tokens_fn is not supported for continuous bat

Error message

prefix_allowed_tokens_fn is not supported for continuous batching. Got {prefix_allowed_tokens_fn = }

What it means

Error "prefix_allowed_tokens_fn is not supported for continuous batching. Got {prefix_allowed_tokens_fn = }" thrown in huggingface/transformers.

Source

Thrown at src/transformers/generation/utils.py:2416

            # generate_batch expects a list of lists of ints, so we create it from the inputs or input_ids
            inputs = inputs if inputs is not None else kwargs.get("input_ids")
            if inputs is None:
                raise ValueError("inputs or input_ids must be provided for CB generation.")

            if inputs.dim() == 1:
                inputs = inputs.unsqueeze(0).tolist()
            elif inputs.dim() == 2:
                inputs = inputs.tolist()
            else:
                raise ValueError(f"inputs must be a 1D or 2D tensor, got {inputs.dim() = }")

            # some arguments are not supported for continuous batching
            if stopping_criteria is not None:
                raise NotImplementedError(
                    f"stopping_criteria is not supported for continuous batching. Got {stopping_criteria = }"
                )
            if prefix_allowed_tokens_fn is not None:
                raise NotImplementedError(
                    f"prefix_allowed_tokens_fn is not supported for continuous batching. Got {prefix_allowed_tokens_fn = }"
                )
            if assistant_model is not None:
                raise NotImplementedError(
                    f"assistant_model is not supported for continuous batching. Got {assistant_model = }"
                )
            if streamer is not None:  # TODO: actually this could be supported
                raise NotImplementedError(f"streaming is not supported for continuous batching. Got {streamer = }")
            if negative_prompt_ids is not None:
                raise NotImplementedError(
                    f"negative_prompt_ids is not supported for continuous batching. Got {negative_prompt_ids = }"
                )
            if negative_prompt_attention_mask is not None:
                raise NotImplementedError(
                    f"negative_prompt_attention_mask is not supported for continuous batching. Got {negative_prompt_attention_mask = }"
                )

            # others are ignored

View on GitHub (pinned to a597f97485)

Solutions

1. Remove `prefix_allowed_tokens_fn` when generating with continuous batching (do_cb=True).
2. Run sequential generation instead if you need prefix constraints.

When it happens

Trigger: Raised in generate() when prefix_allowed_tokens_fn is passed while using continuous batching mode.

Common situations: Structured/constrained decoding arguments combined with the continuous batching generate path, which does not support them.

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/983ae337e8c7d8b9. Report an issue: GitHub.