huggingface/transformers · error · NotImplementedError

streaming is not supported for continuous batching. Got {str

Error message

streaming is not supported for continuous batching. Got {streamer = }

What it means

Error "streaming is not supported for continuous batching. Got {streamer = }" thrown in huggingface/transformers.

Source

Thrown at src/transformers/generation/utils.py:2424

                inputs = inputs.tolist()
            else:
                raise ValueError(f"inputs must be a 1D or 2D tensor, got {inputs.dim() = }")

            # some arguments are not supported for continuous batching
            if stopping_criteria is not None:
                raise NotImplementedError(
                    f"stopping_criteria is not supported for continuous batching. Got {stopping_criteria = }"
                )
            if prefix_allowed_tokens_fn is not None:
                raise NotImplementedError(
                    f"prefix_allowed_tokens_fn is not supported for continuous batching. Got {prefix_allowed_tokens_fn = }"
                )
            if assistant_model is not None:
                raise NotImplementedError(
                    f"assistant_model is not supported for continuous batching. Got {assistant_model = }"
                )
            if streamer is not None:  # TODO: actually this could be supported
                raise NotImplementedError(f"streaming is not supported for continuous batching. Got {streamer = }")
            if negative_prompt_ids is not None:
                raise NotImplementedError(
                    f"negative_prompt_ids is not supported for continuous batching. Got {negative_prompt_ids = }"
                )
            if negative_prompt_attention_mask is not None:
                raise NotImplementedError(
                    f"negative_prompt_attention_mask is not supported for continuous batching. Got {negative_prompt_attention_mask = }"
                )

            # others are ignored
            if synced_gpus is not None:
                logger.warning(f"synced_gpus is ignored for continuous batching. Got {synced_gpus = }")
            num_beams = kwargs.get("num_beams", 1)
            if num_beams > 1:  # FIXME: remove this once CB supports num_beams (which is planned)
                logger.warning(f"num_beams is not supported for continuous batching yet. Got {num_beams = }. ")

            # switch to CB
            outputs = self.generate_batch(

View on GitHub (pinned to a597f97485)

Solutions

1. Do not pass a `streamer` with continuous batching; consume streamed output via the continuous batching output API instead.
2. Use standard `generate()` if you need a TextStreamer.

When it happens

Trigger: Raised in generate() when a streamer is passed while using continuous batching mode.

Common situations: Passing streamer=TextStreamer(...) to a generate() call routed to continuous batching.

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/01fc38c732401603. Report an issue: GitHub.