huggingface/transformers · error · NotImplementedError

assistant_model is not supported for continuous batching. Go

Error message

assistant_model is not supported for continuous batching. Got {assistant_model = }

What it means

Error "assistant_model is not supported for continuous batching. Got {assistant_model = }" thrown in huggingface/transformers.

Source

Thrown at src/transformers/generation/utils.py:2420

            if inputs.dim() == 1:
                inputs = inputs.unsqueeze(0).tolist()
            elif inputs.dim() == 2:
                inputs = inputs.tolist()
            else:
                raise ValueError(f"inputs must be a 1D or 2D tensor, got {inputs.dim() = }")

            # some arguments are not supported for continuous batching
            if stopping_criteria is not None:
                raise NotImplementedError(
                    f"stopping_criteria is not supported for continuous batching. Got {stopping_criteria = }"
                )
            if prefix_allowed_tokens_fn is not None:
                raise NotImplementedError(
                    f"prefix_allowed_tokens_fn is not supported for continuous batching. Got {prefix_allowed_tokens_fn = }"
                )
            if assistant_model is not None:
                raise NotImplementedError(
                    f"assistant_model is not supported for continuous batching. Got {assistant_model = }"
                )
            if streamer is not None:  # TODO: actually this could be supported
                raise NotImplementedError(f"streaming is not supported for continuous batching. Got {streamer = }")
            if negative_prompt_ids is not None:
                raise NotImplementedError(
                    f"negative_prompt_ids is not supported for continuous batching. Got {negative_prompt_ids = }"
                )
            if negative_prompt_attention_mask is not None:
                raise NotImplementedError(
                    f"negative_prompt_attention_mask is not supported for continuous batching. Got {negative_prompt_attention_mask = }"
                )

            # others are ignored
            if synced_gpus is not None:
                logger.warning(f"synced_gpus is ignored for continuous batching. Got {synced_gpus = }")
            num_beams = kwargs.get("num_beams", 1)
            if num_beams > 1:  # FIXME: remove this once CB supports num_beams (which is planned)

View on GitHub (pinned to a597f97485)

Solutions

1. Remove `assistant_model` when using continuous batching.
2. Use standard `generate()` without continuous batching for assisted decoding.

When it happens

Trigger: Raised in generate() when assistant_model is passed while using continuous batching mode.

Common situations: Attempting speculative/assisted decoding together with continuous batching, which is unsupported.

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/ae187c374bc3d9e6. Report an issue: GitHub.