{"record":{"id":"f5eb2fb0db7e5cea","repo":"vllm-project/vllm","slug":"max-num-scheduled-tokens-is-set-to-self-scheduler","errorCode":null,"errorMessage":"max_num_scheduled_tokens is set to {self.scheduler_config.max_num_scheduled_tokens} based on the speculative decoding settings, which does not allow any tokens to be scheduled. Increase max_num_batched_tokens to accommodate the additional draft token slots, or decrease num_speculative_tokens.","messagePattern":"max_num_scheduled_tokens is set to (.+?) based on the speculative decoding settings, which does not allow any tokens to be scheduled\\. Increase max_num_batched_tokens to accommodate the additional draft token slots, or decrease num_speculative_tokens\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"vllm/config/vllm.py","lineNumber":1854,"sourceCode":"            for size in possible_sizes\n            if size % self.parallel_config.tensor_parallel_size == 0\n        ]\n\n    def _set_max_num_scheduled_tokens(self):\n        \"\"\"\n        In most cases, the scheduler may schedule a batch with as many tokens as the\n        worker is configured to handle.\n        \"\"\"\n        if self.speculative_config is not None:\n            scheduled_token_delta = (\n                self.speculative_config.max_num_new_slots_for_drafting\n            )\n            max_num_batched_tokens = self.scheduler_config.max_num_batched_tokens\n            if self.scheduler_config.max_num_scheduled_tokens is None:\n                self.scheduler_config.max_num_scheduled_tokens = max_num_batched_tokens\n\n            if self.scheduler_config.max_num_scheduled_tokens <= 0:\n                raise ValueError(\n                    \"max_num_scheduled_tokens is set to\"\n                    f\" {self.scheduler_config.max_num_scheduled_tokens} based on\"\n                    \" the speculative decoding settings, which does not allow\"\n                    \" any tokens to be scheduled. Increase max_num_batched_tokens\"\n                    \" to accommodate the additional draft token slots, or decrease\"\n                    \" num_speculative_tokens.\"\n                )\n            if self.scheduler_config.max_num_scheduled_tokens < 8192:\n                logger.warning_once(\n                    \"max_num_scheduled_tokens is set to\"\n                    f\" {self.scheduler_config.max_num_scheduled_tokens} based on\"\n                    \" the speculative decoding settings. This may lead to suboptimal\"\n                    \" performance. Consider increasing max_num_batched_tokens to\"\n                    \" accommodate the additional draft token slots, or decrease\"\n                    \" num_speculative_tokens.\",\n                )\n\n            if max_num_batched_tokens <= scheduled_token_delta:","sourceCodeStart":1836,"sourceCodeEnd":1872,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/config/vllm.py#L1836-L1872","documentation":"With speculative decoding, vLLM reserves extra scheduling slots for draft tokens: `max_num_scheduled_tokens` is derived from `max_num_batched_tokens` minus `max_num_new_slots_for_drafting`. If that subtraction lands at zero or below, no token can ever be scheduled, so validation raises. See `_set_max_num_scheduled_tokens` in vllm/config/vllm.py:1840.","triggerScenarios":"Launching with a small `--max-num-batched-tokens` (e.g. equal to or smaller than the draft slot reservation, such as num_speculative_tokens+1 per request) together with a speculative config.","commonSituations":"Tuning batched tokens down for memory on small GPUs while running EAGLE/NGRAM drafting; raising num_speculative_tokens without scaling max_num_batched_tokens.","solutions":["Increase `--max-num-batched-tokens` so it comfortably exceeds the draft-token reservation (e.g. 8192+delta).","Or decrease `num_speculative_tokens` in the speculative config.","Recompute: ensure max_num_batched_tokens > speculative_config.max_num_new_slots_for_drafting."],"exampleFix":"# before\nvllm serve model --max-num-batched-tokens 64 \\\n  --speculative-config '{\"method\":\"eagle\",\"num_speculative_tokens\":8}'\n\n# after\nvllm serve model --max-num-batched-tokens 8192 \\\n  --speculative-config '{\"method\":\"eagle\",\"num_speculative_tokens\":8}'","handlingStrategy":"validation","validationCode":"from vllm.config.vllm import VllmConfig  # or recompute delta\n# rule of thumb: max_num_batched_tokens must exceed draft reservation\nif spec_config and max_num_batched_tokens <= spec_config.num_speculative_tokens + 2:\n    raise SystemExit(\"increase --max-num-batched-tokens or lower num_speculative_tokens\")","typeGuard":null,"tryCatchPattern":"try:\n    LLM(**args)\nexcept ValueError as e:\n    if \"max_num_scheduled_tokens\" in str(e):\n        args[\"max_num_batched_tokens\"] = max(8192, args[\"max_num_batched_tokens\"] * 2)\n    else:\n        raise","preventionTips":["Keep max_num_batched_tokens >= 8192 with speculative decoding","Scale batched tokens when raising num_speculative_tokens"],"tags":["scheduler","speculative-decoding","batched-tokens"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}