huggingface/transformers · critical · MemoryError

Memory footprint {} is more than available memory {}

Error message

Memory footprint {} is more than available memory {}

What it means

Thrown by PagedAttentionCache._check_footprint after the cache solves for max_batch_tokens/num_blocks: the memory polynomial evaluated at the chosen (max_batch_tokens, num_blocks) exceeds available_memory. It is a MemoryError raised during cache auto-sizing (get_max_batch_tokens_and_num_blocks), meaning the requested batch capacity physically cannot fit in the memory budget computed from the device.

Source

Thrown at src/transformers/generation/continuous_batching/cache.py:734

        # Otherwise, use a linear solver
        elif num_blocks is None:
            # M given → linear in N: (coeff_n + coeff_nm·M)·N = avail − coeff_m·M − coeff_mm·M²
            M = max_batch_tokens
            num_pages = floor((self.available_memory - cm * M - cmm * M**2) / (cn + cmn * M))
            num_blocks = num_pages // self.block_size

        elif max_batch_tokens is None:
            # N given → quadratic in M: coeff_mm·M² + (coeff_m + coeff_nm·N)·M + (coeff_n·N − avail) = 0
            N = num_blocks * self.block_size
            max_batch_tokens = int(self._solve_quadratic(cmm, cm + cmn * N, cn * N - self.available_memory))

        return max_batch_tokens, num_blocks

    def _check_footprint(self, max_batch_tokens: int, num_blocks: int) -> tuple[int, int]:
        """Checks if the footprint of the cache is within the available memory."""
        memory_footprint = self.compute_memory_footprint(max_batch_tokens, num_blocks)
        if memory_footprint > self.available_memory:
            raise MemoryError(
                f"Memory footprint {memory_footprint} is more than available memory {self.available_memory}"
            )
        if max_batch_tokens <= 0 or num_blocks <= 0:
            raise ValueError(f"Invalid values: max_batch_tokens = {max_batch_tokens}, num_blocks = {num_blocks}")
        return max_batch_tokens, num_blocks

    def _solve_quadratic(self, a: float, b: float, c: float) -> int:
        """Largest positive root of a·x² + b·x + c = 0. Falls back to linear when a == 0. Rounded down."""
        if a == 0:
            return int(-c / b)
        discriminant = b**2 - 4 * a * c
        if discriminant < 0:
            raise ValueError(f"No real solution (discriminant = {discriminant})")
        root = (-b + sqrt(discriminant)) / (2 * a)
        if root < 0:
            raise ValueError(f"No positive solution (root = {root})")
        return int(floor(root))

View on GitHub (pinned to a597f97485)

Solutions

1. Lower max_batch_tokens (or num_blocks) when constructing the ContinuousBatchingConfig / cache so the footprint fits available_memory
2. Reduce cache/activation memory: use a smaller cache dtype (e.g. float16/bfloat16 KV cache), a smaller model, or fewer layers
3. Free memory before manager creation: del previous managers/tensors and torch.cuda.empty_cache(), since available_memory is measured at setup time
4. If auto-sizing triggered it, report upstream — auto-sizing should never exceed the budget, so a rounding/ordering bug may exist in the solver

Example fix

# before
config = ContinuousBatchingConfig(max_batch_tokens=32768)  # too large for the GPU

# after
config = ContinuousBatchingConfig(max_batch_tokens=4096)  # fit to available memory

free_b, total_b = torch.cuda.mem_get_info() if torch.cuda.is_available() else (psutil.virtual_memory().available, psutil.virtual_memory().total)
# footprint grows with max_batch_tokens; start conservative and scale up
assert free_b > 2 * 1024**3, f"only {free_b/1024**3:.1f} GiB free — free memory before starting"

try:
    manager = model.continuous_batching(config=cfg)
except MemoryError as e:
    cfg.max_batch_tokens = max(256, cfg.max_batch_tokens // 2)
    torch.cuda.empty_cache()
    manager = model.continuous_batching(config=cfg)

When it happens

Trigger: Calling the continuous-batching setup path that ends in _check_footprint (e.g. ContinuousBatchingManager creation) with an explicit max_batch_tokens or num_blocks whose footprint (activation peaks + KV cache) exceeds available_memory; or auto-sizing when available_memory was computed too optimistically (other tensors already allocated, small GPU).

Common situations: Small GPU (e.g. 8GB) with a large model or fp32 cache; leftover allocations from a previous run shrinking available_memory; user passes max_batch_tokens copied from a bigger-GPU setup; quantization/cache dtype not applied so cache bytes are 2x expected.

Related errors

• Failed to allocate {} blocks for request {}
• Invalid values: max_batch_tokens = {}, num_blocks = {}
• num_key_value_heads or num_attention_heads could not be foun
• head_dim or (hidden_size and num_attention_heads) could not
• Block size must be at least {}, but got {}

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/e658498444bd4dba. Report an issue: GitHub.