{"record":{"id":"ae187c374bc3d9e6","repo":"huggingface/transformers","slug":"assistant-model-is-not-supported-for-continuous-ba","errorCode":null,"errorMessage":"assistant_model is not supported for continuous batching. Got {assistant_model = }","messagePattern":"assistant_model is not supported for continuous batching\\. Got (.+?)","errorType":"validation","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"src/transformers/generation/utils.py","lineNumber":2420,"sourceCode":"\n            if inputs.dim() == 1:\n                inputs = inputs.unsqueeze(0).tolist()\n            elif inputs.dim() == 2:\n                inputs = inputs.tolist()\n            else:\n                raise ValueError(f\"inputs must be a 1D or 2D tensor, got {inputs.dim() = }\")\n\n            # some arguments are not supported for continuous batching\n            if stopping_criteria is not None:\n                raise NotImplementedError(\n                    f\"stopping_criteria is not supported for continuous batching. Got {stopping_criteria = }\"\n                )\n            if prefix_allowed_tokens_fn is not None:\n                raise NotImplementedError(\n                    f\"prefix_allowed_tokens_fn is not supported for continuous batching. Got {prefix_allowed_tokens_fn = }\"\n                )\n            if assistant_model is not None:\n                raise NotImplementedError(\n                    f\"assistant_model is not supported for continuous batching. Got {assistant_model = }\"\n                )\n            if streamer is not None:  # TODO: actually this could be supported\n                raise NotImplementedError(f\"streaming is not supported for continuous batching. Got {streamer = }\")\n            if negative_prompt_ids is not None:\n                raise NotImplementedError(\n                    f\"negative_prompt_ids is not supported for continuous batching. Got {negative_prompt_ids = }\"\n                )\n            if negative_prompt_attention_mask is not None:\n                raise NotImplementedError(\n                    f\"negative_prompt_attention_mask is not supported for continuous batching. Got {negative_prompt_attention_mask = }\"\n                )\n\n            # others are ignored\n            if synced_gpus is not None:\n                logger.warning(f\"synced_gpus is ignored for continuous batching. Got {synced_gpus = }\")\n            num_beams = kwargs.get(\"num_beams\", 1)\n            if num_beams > 1:  # FIXME: remove this once CB supports num_beams (which is planned)","sourceCodeStart":2402,"sourceCodeEnd":2438,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/generation/utils.py#L2402-L2438","documentation":"Error \"assistant_model is not supported for continuous batching. Got {assistant_model = }\" thrown in huggingface/transformers.","triggerScenarios":"Raised in generate() when assistant_model is passed while using continuous batching mode.","commonSituations":"Attempting speculative/assisted decoding together with continuous batching, which is unsupported.","solutions":["Remove `assistant_model` when using continuous batching.","Use standard `generate()` without continuous batching for assisted decoding."],"exampleFix":null,"handlingStrategy":null,"validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":[],"tags":[],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}