huggingface/transformers · error · ValueError

TextStreamer only supports batch size 1

Error message

TextStreamer only supports batch size 1

What it means

ValueError from TextStreamer.put: streaming generation to stdout only supports batch size 1, because tokens are decoded incrementally into a single rolling text cache and printed in order. A batch dimension > 1 has no meaningful interleaved printing, so it is rejected.

Source

Thrown at src/transformers/generation/streamers.py:85

        ```
    """

    def __init__(self, tokenizer: PreTrainedTokenizerBase, skip_prompt: bool = False, **decode_kwargs: Any):
        self.tokenizer = tokenizer
        self.skip_prompt = skip_prompt
        self.decode_kwargs = decode_kwargs

        # variables used in the streaming process
        self.token_cache: list[int] = []
        self.print_len = 0
        self.next_tokens_are_prompt = True

    def put(self, value):
        """
        Receives tokens, decodes them, and prints them to stdout as soon as they form entire words.
        """
        if len(value.shape) > 1 and value.shape[0] > 1:
            raise ValueError("TextStreamer only supports batch size 1")
        elif len(value.shape) > 1:
            value = value[0]

        if self.skip_prompt and self.next_tokens_are_prompt:
            self.next_tokens_are_prompt = False
            return

        # Add the new token to the cache and decodes the entire thing.
        self.token_cache.extend(value.tolist())
        text = cast(str, self.tokenizer.decode(self.token_cache, **self.decode_kwargs))

        # After the symbol for a new line, we flush the cache.
        if text.endswith("\n"):
            printable_text = text[self.print_len :]
            self.token_cache = []
            self.print_len = 0
        # If the last token is a CJK character, we print the characters.
        elif len(text) > 0 and self._is_chinese_char(ord(text[-1])):

View on GitHub (pinned to a597f97485)

Solutions

1. Run generation one sequence at a time (batch size 1) when a streamer is attached.
2. Remove the streamer for batched runs and decode afterwards.
3. Use a custom streamer subclass that buffers per batch index instead of printing.

Example fix

# before
out = model.generate(**tokenizer([p1, p2], return_tensors="pt", padding=True), streamer=TextStreamer(tok))

# after
for p in [p1, p2]:
    out = model.generate(**tokenizer(p, return_tensors="pt"), streamer=TextStreamer(tok))

if streamer is not None and inputs.input_ids.shape[0] > 1:
    raise ValueError("detach streamer or use batch size 1")

When it happens

Trigger: model.generate(batched_inputs, streamer=TextStreamer(tokenizer)) with input_ids.shape[0] > 1; padding two prompts into one batch and passing the streamer.

Common situations: Batch inference scripts retrofitted with a streamer for a demo; serving code that reuses one generate call for multiple requests while also streaming.

Related errors

• TextDiffusionStreamer only supports batch size 1
• `streamer` cannot be used with beam search (yet!). Make sure
• `crop` was called, but the current layer does not track past
• Once the sliding window size has been reached, `DynamicSlidi
• `crop` was called, but the current layer does not track past

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/ec34e1dc12305a2a. Report an issue: GitHub.