huggingface/transformers · error · NotImplementedError

stopping_criteria is not supported for continuous batching.

Error message

stopping_criteria is not supported for continuous batching. Got {stopping_criteria = }

What it means

The continuous-batching path (`cache_implementation="paged"` -> `generate_batch`) has its own stopping logic and does not accept a user `StoppingCriteriaList`. During the switch, generate explicitly rejects call-time `stopping_criteria` (a NotImplementedError) rather than silently ignoring your criteria.

Source

Thrown at src/transformers/generation/utils.py:2412

                "Detected cache_implementation=paged: switching to continuous batching. You should consider using "
                "generate_batch directly instead."
            )

            # generate_batch expects a list of lists of ints, so we create it from the inputs or input_ids
            inputs = inputs if inputs is not None else kwargs.get("input_ids")
            if inputs is None:
                raise ValueError("inputs or input_ids must be provided for CB generation.")

            if inputs.dim() == 1:
                inputs = inputs.unsqueeze(0).tolist()
            elif inputs.dim() == 2:
                inputs = inputs.tolist()
            else:
                raise ValueError(f"inputs must be a 1D or 2D tensor, got {inputs.dim() = }")

            # some arguments are not supported for continuous batching
            if stopping_criteria is not None:
                raise NotImplementedError(
                    f"stopping_criteria is not supported for continuous batching. Got {stopping_criteria = }"
                )
            if prefix_allowed_tokens_fn is not None:
                raise NotImplementedError(
                    f"prefix_allowed_tokens_fn is not supported for continuous batching. Got {prefix_allowed_tokens_fn = }"
                )
            if assistant_model is not None:
                raise NotImplementedError(
                    f"assistant_model is not supported for continuous batching. Got {assistant_model = }"
                )
            if streamer is not None:  # TODO: actually this could be supported
                raise NotImplementedError(f"streaming is not supported for continuous batching. Got {streamer = }")
            if negative_prompt_ids is not None:
                raise NotImplementedError(
                    f"negative_prompt_ids is not supported for continuous batching. Got {negative_prompt_ids = }"
                )
            if negative_prompt_attention_mask is not None:
                raise NotImplementedError(

View on GitHub (pinned to a597f97485)

Solutions

1. Remove `stopping_criteria` from the call and express stopping via supported args, e.g. `max_new_tokens` or `stop_strings` + `tokenizer`,
2. or stay on the standard generation path (`cache_implementation=None/"dynamic"`) where `stopping_criteria` works.
3. If you need custom criteria AND continuous batching, implement stopping at the application layer: run `generate_batch` in a loop and stop consuming/cancel when your condition is met.
4. Don't set `cache_implementation="paged"` as a global default if any caller relies on custom stopping criteria.

Example fix

# before
out = model.generate(**inputs, cache_implementation="paged", stopping_criteria=StoppingCriteriaList([max_criteria]))  # NotImplementedError

# after
out = model.generate(**inputs, cache_implementation="paged", max_new_tokens=50, stop_strings=["\n\n"], tokenizer=tokenizer)

if kwargs.get("cache_implementation") == "paged" and kwargs.get("stopping_criteria") is not None:
    kwargs.pop("stopping_criteria")
    kwargs.setdefault("max_new_tokens", 64)  # express stopping via supported args

When it happens

Trigger: `model.generate(**inputs, cache_implementation="paged", stopping_criteria=StoppingCriteriaList([MaxLengthCriteria(50)]))` or any paged-cache call where `stopping_criteria` was passed positionally/programmatically.

Common situations: Migrating existing generate pipelines that use custom stopping criteria (stop on regex, token budget, external signals) to paged KV caches; serving stacks that always inject a `StoppingCriteriaList`; setting `cache_implementation` globally so criteria-carrying calls break.

Related errors

• inputs or input_ids must be provided for CB generation.
• inputs must be a 1D or 2D tensor, got {inputs.dim() = }
• StoppingCriteria needs to be subclassed
• Stop string preprocessing was unable to identify tokens matc
• There are one or more stop strings, either in the arguments

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/35c1d7a3c27084df. Report an issue: GitHub.