huggingface/transformers · error · ValueError

Got {max_requests_per_batch = } but expected a value >= 1

Error message

Got {max_requests_per_batch = } but expected a value >= 1

What it means

Scheduler.__init__ validates max_requests_per_batch >= 1: zero or negative batch sizes are meaningless and rejected. This bounds how many requests share one scheduler step.

Source

Thrown at src/transformers/generation/continuous_batching/scheduler.py:42

    Abstract base class for scheduling requests in the continuous batch processor. Schedulers manage the lifecycle of
    requests from when they are added to the waiting queue to when they are scheduled for processing. Different
    schedulers implement different strategies for prioritizing and batching requests.
    """

    def __init__(self, cache: PagedAttentionCache, safety_margin: float, max_requests_per_batch: int):
        """Initializes the scheduler. The safety margin is the percentage of free blocks under which we stop
        scheduling new prefill requests, so safety_margin = 0.1 means that when there is less than 10% of free blocks,
        or equivalently when more than 90% of blocks are already allocated, we stop scheduling new prefill requests.
        Setting safety_margin to 0.0 means no safety margin is applied."""
        self.cache = cache
        self.safety_margin = safety_margin
        self.max_requests_per_batch = max_requests_per_batch
        self._cancellation_lock = threading.Lock()
        # Check args
        if safety_margin < 0 or safety_margin > 1:
            raise ValueError(f"Got {safety_margin = } but expected a value in [0, 1]")
        if max_requests_per_batch < 1:
            raise ValueError(f"Got {max_requests_per_batch = } but expected a value >= 1")
        # This is to compute the read cache used by a new request being scheduled
        self.read_cache_limit = None if self.cache.num_full_attention_groups else self.cache.config.sliding_window
        self.max_decode_fast_path_length = self.cache.max_blocks_per_request * self.cache.block_size
        # Initialize mutable states via reset()
        self.reset()

    def reset(self) -> None:
        """Reset scheduler state for a new generation loop."""
        self.active_requests: dict[str, RequestState] = {}
        self.waiting_requests: dict[str, RequestState] = {}
        self.waiting_requests_order: deque[str] = deque()
        self._requests_to_cancel: set[str] = set()
        self._requests_to_fork: list[RequestState] = []
        self.block_new_requests = False
        # Active requests that failed block allocation in the last scheduled batch, with their physical block demand.
        # The offloading manager uses this to size bulk evictions.
        self.starved_requests: list[tuple[RequestState, int]] = []

View on GitHub (pinned to a597f97485)

Solutions

1. Set max_requests_per_batch to at least 1 (typically 16-256 depending on hardware)
2. If the value is computed, clamp it: max(1, computed_value)
3. Audit env-var parsing that can yield 0 for unset variables

Example fix

# before
n = int(os.environ.get('MAX_REQ', 0))
cfg = ContinuousBatchingConfig(max_requests_per_batch=n)

# after
n = max(1, int(os.environ.get('MAX_REQ', 32)))
cfg = ContinuousBatchingConfig(max_requests_per_batch=n)

cfg.max_requests_per_batch = max(1, int(cfg.max_requests_per_batch))

def is_valid_max_requests(v) -> bool:
    return isinstance(v, int) and v >= 1

When it happens

Trigger: Passing max_requests_per_batch=0 in ContinuousBatchingConfig — commonly from a computed value (num_gpus - 1 with 1 GPU, a config default of 0, or math gone wrong).

Common situations: Dynamic configs computing the value from environment variables (empty string parsed as 0); scaling experiments sweeping to 0; copy of a template with a placeholder 0.

Related errors

• Got {safety_margin = } but expected a value in [0, 1]
• m must be provided if max_batch_tokens and num_blocks are No
• db_range must be greater than zero
• Stage_names must be set for transformers backbones
• out_features must be a list got {type(self._out_features)}

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/7790d620c26d7671. Report an issue: GitHub.