{"record":{"id":"dc61c611dc5e39d5","repo":"huggingface/transformers","slug":"inputs-or-input-ids-must-be-provided-for-cb-genera","errorCode":null,"errorMessage":"inputs or input_ids must be provided for CB generation.","messagePattern":"inputs or input_ids must be provided for CB generation\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/generation/utils.py","lineNumber":2401,"sourceCode":"            generate_arguments = {key: value for key, value in locals().items() if key not in global_keys_to_exclude}\n            generate_arguments.update(kwargs)\n\n            custom_generate_function = self.load_custom_generate(\n                custom_generate, trust_remote_code=trust_remote_code, **kwargs\n            )\n            return custom_generate_function(model=self, **generate_arguments)\n\n        # 0.b. If requested, switched to continuous batching generation\n        if kwargs.get(\"cache_implementation\") == \"paged\":\n            logger.warning(\n                \"Detected cache_implementation=paged: switching to continuous batching. You should consider using \"\n                \"generate_batch directly instead.\"\n            )\n\n            # generate_batch expects a list of lists of ints, so we create it from the inputs or input_ids\n            inputs = inputs if inputs is not None else kwargs.get(\"input_ids\")\n            if inputs is None:\n                raise ValueError(\"inputs or input_ids must be provided for CB generation.\")\n\n            if inputs.dim() == 1:\n                inputs = inputs.unsqueeze(0).tolist()\n            elif inputs.dim() == 2:\n                inputs = inputs.tolist()\n            else:\n                raise ValueError(f\"inputs must be a 1D or 2D tensor, got {inputs.dim() = }\")\n\n            # some arguments are not supported for continuous batching\n            if stopping_criteria is not None:\n                raise NotImplementedError(\n                    f\"stopping_criteria is not supported for continuous batching. Got {stopping_criteria = }\"\n                )\n            if prefix_allowed_tokens_fn is not None:\n                raise NotImplementedError(\n                    f\"prefix_allowed_tokens_fn is not supported for continuous batching. Got {prefix_allowed_tokens_fn = }\"\n                )\n            if assistant_model is not None:","sourceCodeStart":2383,"sourceCodeEnd":2419,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/generation/utils.py#L2383-L2419","documentation":"`cache_implementation=\"paged\"` makes `generate` switch to the continuous-batching backend (`generate_batch`), which needs the raw prompts: it builds a list-of-lists of token ids from `inputs` or `kwargs[\"input_ids\"]`. If both are None there is nothing to schedule, so it raises before switching.","triggerScenarios":"`model.generate(cache_implementation=\"paged\")` with no positional `inputs` and no `input_ids` in kwargs — e.g. relying on `decoder_start_token_id`-only decoding, or a wrapper that strips inputs.","commonSituations":"Setting `cache_implementation` globally (model.generation_config or a serving default) so some calls that legitimately pass no inputs now route into continuous batching; refactors renaming `input_ids` before generate; unconditional KV-paged settings in inference servers.","solutions":["Pass the prompts: `model.generate(inputs=tokenizer(prompt, return_tensors=\"pt\").input_ids, cache_implementation=\"paged')` or include `input_ids` in the call.","Only enable `cache_implementation=\"paged'` on calls that actually carry inputs; don't set it as a blanket default.","Prefer calling `generate_batch` directly for continuous batching, as the warning suggests."],"exampleFix":"# before\nout = model.generate(cache_implementation=\"paged\")  # no inputs -> ValueError\n\n# after\nencoded = tokenizer(prompt, return_tensors=\"pt\")\nout = model.generate(**encoded, cache_implementation=\"paged\")","handlingStrategy":"validation","validationCode":"if kwargs.get(\"cache_implementation\") == \"paged\" and inputs is None and kwargs.get(\"input_ids\") is None:\n    raise ValueError(\"continuous batching requires inputs or input_ids\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Only set cache_implementation='paged' on calls that carry prompts.","Call generate_batch directly for continuous batching instead of the implicit switch.","Keep a plain-generation path for prompt-less decoding."],"tags":["generation","continuous-batching","paged-cache","missing-input"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}