huggingface/transformers · error · ValueError

inputs or input_ids must be provided for CB generation.

Error message

inputs or input_ids must be provided for CB generation.

What it means

`cache_implementation="paged"` makes `generate` switch to the continuous-batching backend (`generate_batch`), which needs the raw prompts: it builds a list-of-lists of token ids from `inputs` or `kwargs["input_ids"]`. If both are None there is nothing to schedule, so it raises before switching.

Source

Thrown at src/transformers/generation/utils.py:2401

            generate_arguments = {key: value for key, value in locals().items() if key not in global_keys_to_exclude}
            generate_arguments.update(kwargs)

            custom_generate_function = self.load_custom_generate(
                custom_generate, trust_remote_code=trust_remote_code, **kwargs
            )
            return custom_generate_function(model=self, **generate_arguments)

        # 0.b. If requested, switched to continuous batching generation
        if kwargs.get("cache_implementation") == "paged":
            logger.warning(
                "Detected cache_implementation=paged: switching to continuous batching. You should consider using "
                "generate_batch directly instead."
            )

            # generate_batch expects a list of lists of ints, so we create it from the inputs or input_ids
            inputs = inputs if inputs is not None else kwargs.get("input_ids")
            if inputs is None:
                raise ValueError("inputs or input_ids must be provided for CB generation.")

            if inputs.dim() == 1:
                inputs = inputs.unsqueeze(0).tolist()
            elif inputs.dim() == 2:
                inputs = inputs.tolist()
            else:
                raise ValueError(f"inputs must be a 1D or 2D tensor, got {inputs.dim() = }")

            # some arguments are not supported for continuous batching
            if stopping_criteria is not None:
                raise NotImplementedError(
                    f"stopping_criteria is not supported for continuous batching. Got {stopping_criteria = }"
                )
            if prefix_allowed_tokens_fn is not None:
                raise NotImplementedError(
                    f"prefix_allowed_tokens_fn is not supported for continuous batching. Got {prefix_allowed_tokens_fn = }"
                )
            if assistant_model is not None:

View on GitHub (pinned to a597f97485)

Solutions

1. Pass the prompts: `model.generate(inputs=tokenizer(prompt, return_tensors="pt").input_ids, cache_implementation="paged')` or include `input_ids` in the call.
2. Only enable `cache_implementation="paged'` on calls that actually carry inputs; don't set it as a blanket default.
3. Prefer calling `generate_batch` directly for continuous batching, as the warning suggests.

Example fix

# before
out = model.generate(cache_implementation="paged")  # no inputs -> ValueError

# after
encoded = tokenizer(prompt, return_tensors="pt")
out = model.generate(**encoded, cache_implementation="paged")

if kwargs.get("cache_implementation") == "paged" and inputs is None and kwargs.get("input_ids") is None:
    raise ValueError("continuous batching requires inputs or input_ids")

When it happens

Trigger: `model.generate(cache_implementation="paged")` with no positional `inputs` and no `input_ids` in kwargs — e.g. relying on `decoder_start_token_id`-only decoding, or a wrapper that strips inputs.

Common situations: Setting `cache_implementation` globally (model.generation_config or a serving default) so some calls that legitimately pass no inputs now route into continuous batching; refactors renaming `input_ids` before generate; unconditional KV-paged settings in inference servers.

Related errors

• inputs must be a 1D or 2D tensor, got {inputs.dim() = }
• stopping_criteria is not supported for continuous batching.
• `crop` was called, but the current layer does not track past
• Once the sliding window size has been reached, `DynamicSlidi
• `crop` was called, but the current layer does not track past

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/dc61c611dc5e39d5. Report an issue: GitHub.