huggingface/transformers · error · ValueError

m must be provided if max_batch_tokens and num_blocks are No

Error message

m must be provided if max_batch_tokens and num_blocks are None

What it means

ValueError from PagedAttentionMemoryHandler's solver: when both max_batch_tokens and num_blocks are None, the memory equation is closed by substituting M = m*N (max_batch_tokens = cache_fill_per_batch * num_blocks); without a ratio m the quadratic has two unknowns and cannot be solved. cache_fill_per_batch encodes the target ratio between activation memory and KV cache memory.

Source

Thrown at src/transformers/generation/continuous_batching/cache.py:710

        final_n = min([solution[1] for solution in solutions])
        return final_m, final_n

    def _solve_for_peak(
        self,
        peak: tuple[int, ...],
        max_batch_tokens: int | None,
        num_blocks: int | None,
        cache_fill_per_batch: float | None,
    ) -> tuple[int, int]:
        """Returns a couple of `(max_batch_tokens, num_blocks)` that satisfy the memory constraint for the given
        activation peak."""
        cm, cn, cmn, cmm = self._equation_coefficients(peak)

        # If neither variable is defined, use a quadratic solver
        if max_batch_tokens is None and num_blocks is None:
            # Substitute M = m·N → (coeff_nm·m + coeff_mm·m²)·N² + (coeff_n + coeff_m·m)·N − avail = 0
            if cache_fill_per_batch is None:
                raise ValueError("m must be provided if max_batch_tokens and num_blocks are None")
            m = cache_fill_per_batch  # as in, m is a substitute for big M, which is max_batch_tokens
            num_pages = self._solve_quadratic(cmn * m + cmm * m**2, cn + cm * m, -self.available_memory)
            max_batch_tokens = int(num_pages * m)
            num_blocks = int(num_pages) // self.block_size

        # Otherwise, use a linear solver
        elif num_blocks is None:
            # M given → linear in N: (coeff_n + coeff_nm·M)·N = avail − coeff_m·M − coeff_mm·M²
            M = max_batch_tokens
            num_pages = floor((self.available_memory - cm * M - cmm * M**2) / (cn + cmn * M))
            num_blocks = num_pages // self.block_size

        elif max_batch_tokens is None:
            # N given → quadratic in M: coeff_mm·M² + (coeff_m + coeff_nm·N)·M + (coeff_n·N − avail) = 0
            N = num_blocks * self.block_size
            max_batch_tokens = int(self._solve_quadratic(cmm, cm + cmn * N, cn * N - self.available_memory))

        return max_batch_tokens, num_blocks

View on GitHub (pinned to a597f97485)

Solutions

1. Set cache_fill_per_batch (the m ratio) so the solver can size the cache, e.g. ContinuousBatchingConfig(cache_fill_per_batch=8)
2. Or provide max_batch_tokens and let the solver derive num_blocks linearly
3. Or provide num_blocks directly and let max_batch_tokens be derived

Example fix

# before
cb_cfg = ContinuousBatchingConfig(max_batch_tokens=None, num_blocks=None, cache_fill_per_batch=None)
# after
cb_cfg = ContinuousBatchingConfig(cache_fill_per_batch=8)

if cb_cfg.max_batch_tokens is None and cb_cfg.num_blocks is None:
    assert cb_cfg.cache_fill_per_batch is not None, 'set cache_fill_per_batch when auto-sizing memory'
    cb_cfg.cache_fill_per_batch = cb_cfg.cache_fill_per_batch or 8

def memory_plan_is_solvable(cb_cfg) -> bool:
    return cb_cfg.max_batch_tokens is not None or cb_cfg.num_blocks is not None or cb_cfg.cache_fill_per_batch is not None

When it happens

Trigger: Calling the memory planner / resolve_max_memory_percent with both max_batch_tokens=None and num_blocks=None and no cache_fill_per_batch; a ContinuousBatchingConfig where none of the three fields is set and automatic sizing kicks in.

Common situations: Relying on automatic memory sizing on a new setup where the default fill ratio was not propagated; constructing the handler directly in tests or custom servers.

Related errors

• Got {safety_margin = } but expected a value in [0, 1]
• Got {max_requests_per_batch = } but expected a value >= 1
• db_range must be greater than zero
• Stage_names must be set for transformers backbones
• out_features must be a list got {type(self._out_features)}

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/3ecacf75766e2c17. Report an issue: GitHub.