huggingface/transformers · error · ValueError

Got {safety_margin = } but expected a value in [0, 1]

Error message

Got {safety_margin = } but expected a value in [0, 1]

What it means

Scheduler.__init__ validates safety_margin: the fraction of free cache blocks below which new prefill requests stop being scheduled. Values outside [0, 1] are rejected — it is a percentage, not a token count or a byte count.

Source

Thrown at src/transformers/generation/continuous_batching/scheduler.py:40

class Scheduler(ABC):
    """
    Abstract base class for scheduling requests in the continuous batch processor. Schedulers manage the lifecycle of
    requests from when they are added to the waiting queue to when they are scheduled for processing. Different
    schedulers implement different strategies for prioritizing and batching requests.
    """

    def __init__(self, cache: PagedAttentionCache, safety_margin: float, max_requests_per_batch: int):
        """Initializes the scheduler. The safety margin is the percentage of free blocks under which we stop
        scheduling new prefill requests, so safety_margin = 0.1 means that when there is less than 10% of free blocks,
        or equivalently when more than 90% of blocks are already allocated, we stop scheduling new prefill requests.
        Setting safety_margin to 0.0 means no safety margin is applied."""
        self.cache = cache
        self.safety_margin = safety_margin
        self.max_requests_per_batch = max_requests_per_batch
        self._cancellation_lock = threading.Lock()
        # Check args
        if safety_margin < 0 or safety_margin > 1:
            raise ValueError(f"Got {safety_margin = } but expected a value in [0, 1]")
        if max_requests_per_batch < 1:
            raise ValueError(f"Got {max_requests_per_batch = } but expected a value >= 1")
        # This is to compute the read cache used by a new request being scheduled
        self.read_cache_limit = None if self.cache.num_full_attention_groups else self.cache.config.sliding_window
        self.max_decode_fast_path_length = self.cache.max_blocks_per_request * self.cache.block_size
        # Initialize mutable states via reset()
        self.reset()

    def reset(self) -> None:
        """Reset scheduler state for a new generation loop."""
        self.active_requests: dict[str, RequestState] = {}
        self.waiting_requests: dict[str, RequestState] = {}
        self.waiting_requests_order: deque[str] = deque()
        self._requests_to_cancel: set[str] = set()
        self._requests_to_fork: list[RequestState] = []
        self.block_new_requests = False
        # Active requests that failed block allocation in the last scheduled batch, with their physical block demand.
        # The offloading manager uses this to size bulk evictions.

View on GitHub (pinned to a597f97485)

Solutions

1. Express the margin as a fraction in [0, 1]: 0.1 for 10%
2. Use 0.0 only if you deliberately want no safety margin
3. Validate config values after loading from YAML/JSON before passing them in

Example fix

# before
cfg = ContinuousBatchingConfig(safety_margin=10)

# after
cfg = ContinuousBatchingConfig(safety_margin=0.1)

assert isinstance(cfg.safety_margin, (int, float)) and 0 <= cfg.safety_margin <= 1, \
    f'safety_margin must be a fraction in [0,1], got {cfg.safety_margin}'

def is_valid_safety_margin(v) -> bool:
    return isinstance(v, (int, float)) and 0 <= v <= 1

When it happens

Trigger: Constructing the scheduler (via ContinuousBatchingConfig) with safety_margin=10 (user means '10%' but writes 10) or a negative value. 0.1 means stop scheduling when <10% blocks are free.

Common situations: Config files ported from other engines where the margin is expressed in blocks or permille; YAML parsing producing strings/ints >1; mental model mismatch with 'reserved blocks' semantics.

Related errors

• Got {max_requests_per_batch = } but expected a value >= 1
• m must be provided if max_batch_tokens and num_blocks are No
• db_range must be greater than zero
• Stage_names must be set for transformers backbones
• out_features must be a list got {type(self._out_features)}

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/c45d039ea1e96d75. Report an issue: GitHub.