{"record":{"id":"35c1d7a3c27084df","repo":"huggingface/transformers","slug":"stopping-criteria-is-not-supported-for-continuous","errorCode":null,"errorMessage":"stopping_criteria is not supported for continuous batching. Got {stopping_criteria = }","messagePattern":"stopping_criteria is not supported for continuous batching\\. Got (.+?)","errorType":"validation","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"src/transformers/generation/utils.py","lineNumber":2412,"sourceCode":"                \"Detected cache_implementation=paged: switching to continuous batching. You should consider using \"\n                \"generate_batch directly instead.\"\n            )\n\n            # generate_batch expects a list of lists of ints, so we create it from the inputs or input_ids\n            inputs = inputs if inputs is not None else kwargs.get(\"input_ids\")\n            if inputs is None:\n                raise ValueError(\"inputs or input_ids must be provided for CB generation.\")\n\n            if inputs.dim() == 1:\n                inputs = inputs.unsqueeze(0).tolist()\n            elif inputs.dim() == 2:\n                inputs = inputs.tolist()\n            else:\n                raise ValueError(f\"inputs must be a 1D or 2D tensor, got {inputs.dim() = }\")\n\n            # some arguments are not supported for continuous batching\n            if stopping_criteria is not None:\n                raise NotImplementedError(\n                    f\"stopping_criteria is not supported for continuous batching. Got {stopping_criteria = }\"\n                )\n            if prefix_allowed_tokens_fn is not None:\n                raise NotImplementedError(\n                    f\"prefix_allowed_tokens_fn is not supported for continuous batching. Got {prefix_allowed_tokens_fn = }\"\n                )\n            if assistant_model is not None:\n                raise NotImplementedError(\n                    f\"assistant_model is not supported for continuous batching. Got {assistant_model = }\"\n                )\n            if streamer is not None:  # TODO: actually this could be supported\n                raise NotImplementedError(f\"streaming is not supported for continuous batching. Got {streamer = }\")\n            if negative_prompt_ids is not None:\n                raise NotImplementedError(\n                    f\"negative_prompt_ids is not supported for continuous batching. Got {negative_prompt_ids = }\"\n                )\n            if negative_prompt_attention_mask is not None:\n                raise NotImplementedError(","sourceCodeStart":2394,"sourceCodeEnd":2430,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/generation/utils.py#L2394-L2430","documentation":"The continuous-batching path (`cache_implementation=\"paged\"` -> `generate_batch`) has its own stopping logic and does not accept a user `StoppingCriteriaList`. During the switch, generate explicitly rejects call-time `stopping_criteria` (a NotImplementedError) rather than silently ignoring your criteria.","triggerScenarios":"`model.generate(**inputs, cache_implementation=\"paged\", stopping_criteria=StoppingCriteriaList([MaxLengthCriteria(50)]))` or any paged-cache call where `stopping_criteria` was passed positionally/programmatically.","commonSituations":"Migrating existing generate pipelines that use custom stopping criteria (stop on regex, token budget, external signals) to paged KV caches; serving stacks that always inject a `StoppingCriteriaList`; setting `cache_implementation` globally so criteria-carrying calls break.","solutions":["Remove `stopping_criteria` from the call and express stopping via supported args, e.g. `max_new_tokens` or `stop_strings` + `tokenizer`,","or stay on the standard generation path (`cache_implementation=None/\"dynamic\"`) where `stopping_criteria` works.","If you need custom criteria AND continuous batching, implement stopping at the application layer: run `generate_batch` in a loop and stop consuming/cancel when your condition is met.","Don't set `cache_implementation=\"paged\"` as a global default if any caller relies on custom stopping criteria."],"exampleFix":"# before\nout = model.generate(**inputs, cache_implementation=\"paged\", stopping_criteria=StoppingCriteriaList([max_criteria]))  # NotImplementedError\n\n# after\nout = model.generate(**inputs, cache_implementation=\"paged\", max_new_tokens=50, stop_strings=[\"\\n\\n\"], tokenizer=tokenizer)","handlingStrategy":"validation","validationCode":"if kwargs.get(\"cache_implementation\") == \"paged\" and kwargs.get(\"stopping_criteria\") is not None:\n    kwargs.pop(\"stopping_criteria\")\n    kwargs.setdefault(\"max_new_tokens\", 64)  # express stopping via supported args","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Don't pass stopping_criteria when cache_implementation='paged'.","Map criteria to supported equivalents: max_new_tokens, stop_strings + tokenizer.","Implement custom stop conditions in the consuming loop, not inside continuous batching."],"tags":["generation","continuous-batching","paged-cache","stopping-criteria","unsupported-feature"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}