{"record":{"id":"ec34e1dc12305a2a","repo":"huggingface/transformers","slug":"textstreamer-only-supports-batch-size-1","errorCode":null,"errorMessage":"TextStreamer only supports batch size 1","messagePattern":"TextStreamer only supports batch size 1","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/generation/streamers.py","lineNumber":85,"sourceCode":"        ```\n    \"\"\"\n\n    def __init__(self, tokenizer: PreTrainedTokenizerBase, skip_prompt: bool = False, **decode_kwargs: Any):\n        self.tokenizer = tokenizer\n        self.skip_prompt = skip_prompt\n        self.decode_kwargs = decode_kwargs\n\n        # variables used in the streaming process\n        self.token_cache: list[int] = []\n        self.print_len = 0\n        self.next_tokens_are_prompt = True\n\n    def put(self, value):\n        \"\"\"\n        Receives tokens, decodes them, and prints them to stdout as soon as they form entire words.\n        \"\"\"\n        if len(value.shape) > 1 and value.shape[0] > 1:\n            raise ValueError(\"TextStreamer only supports batch size 1\")\n        elif len(value.shape) > 1:\n            value = value[0]\n\n        if self.skip_prompt and self.next_tokens_are_prompt:\n            self.next_tokens_are_prompt = False\n            return\n\n        # Add the new token to the cache and decodes the entire thing.\n        self.token_cache.extend(value.tolist())\n        text = cast(str, self.tokenizer.decode(self.token_cache, **self.decode_kwargs))\n\n        # After the symbol for a new line, we flush the cache.\n        if text.endswith(\"\\n\"):\n            printable_text = text[self.print_len :]\n            self.token_cache = []\n            self.print_len = 0\n        # If the last token is a CJK character, we print the characters.\n        elif len(text) > 0 and self._is_chinese_char(ord(text[-1])):","sourceCodeStart":67,"sourceCodeEnd":103,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/generation/streamers.py#L67-L103","documentation":"ValueError from TextStreamer.put: streaming generation to stdout only supports batch size 1, because tokens are decoded incrementally into a single rolling text cache and printed in order. A batch dimension > 1 has no meaningful interleaved printing, so it is rejected.","triggerScenarios":"model.generate(batched_inputs, streamer=TextStreamer(tokenizer)) with input_ids.shape[0] > 1; padding two prompts into one batch and passing the streamer.","commonSituations":"Batch inference scripts retrofitted with a streamer for a demo; serving code that reuses one generate call for multiple requests while also streaming.","solutions":["Run generation one sequence at a time (batch size 1) when a streamer is attached.","Remove the streamer for batched runs and decode afterwards.","Use a custom streamer subclass that buffers per batch index instead of printing."],"exampleFix":"# before\nout = model.generate(**tokenizer([p1, p2], return_tensors=\"pt\", padding=True), streamer=TextStreamer(tok))\n\n# after\nfor p in [p1, p2]:\n    out = model.generate(**tokenizer(p, return_tensors=\"pt\"), streamer=TextStreamer(tok))","handlingStrategy":"validation","validationCode":"if streamer is not None and inputs.input_ids.shape[0] > 1:\n    raise ValueError(\"detach streamer or use batch size 1\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Gate streaming mode on batch_size == 1 in your CLI/app code.","Keep a batched non-streaming path separate from the interactive streaming path."],"tags":["streamer","batch-size","generation","text-streaming"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}