{"record":{"id":"e658498444bd4dba","repo":"huggingface/transformers","slug":"memory-footprint-is-more-than-available-memory","errorCode":null,"errorMessage":"Memory footprint {} is more than available memory {}","messagePattern":"Memory footprint (.+?) is more than available memory (.+?)","errorType":"exception","errorClass":"MemoryError","httpStatus":null,"severity":"critical","filePath":"src/transformers/generation/continuous_batching/cache.py","lineNumber":734,"sourceCode":"        # Otherwise, use a linear solver\n        elif num_blocks is None:\n            # M given → linear in N: (coeff_n + coeff_nm·M)·N = avail − coeff_m·M − coeff_mm·M²\n            M = max_batch_tokens\n            num_pages = floor((self.available_memory - cm * M - cmm * M**2) / (cn + cmn * M))\n            num_blocks = num_pages // self.block_size\n\n        elif max_batch_tokens is None:\n            # N given → quadratic in M: coeff_mm·M² + (coeff_m + coeff_nm·N)·M + (coeff_n·N − avail) = 0\n            N = num_blocks * self.block_size\n            max_batch_tokens = int(self._solve_quadratic(cmm, cm + cmn * N, cn * N - self.available_memory))\n\n        return max_batch_tokens, num_blocks\n\n    def _check_footprint(self, max_batch_tokens: int, num_blocks: int) -> tuple[int, int]:\n        \"\"\"Checks if the footprint of the cache is within the available memory.\"\"\"\n        memory_footprint = self.compute_memory_footprint(max_batch_tokens, num_blocks)\n        if memory_footprint > self.available_memory:\n            raise MemoryError(\n                f\"Memory footprint {memory_footprint} is more than available memory {self.available_memory}\"\n            )\n        if max_batch_tokens <= 0 or num_blocks <= 0:\n            raise ValueError(f\"Invalid values: max_batch_tokens = {max_batch_tokens}, num_blocks = {num_blocks}\")\n        return max_batch_tokens, num_blocks\n\n    def _solve_quadratic(self, a: float, b: float, c: float) -> int:\n        \"\"\"Largest positive root of a·x² + b·x + c = 0. Falls back to linear when a == 0. Rounded down.\"\"\"\n        if a == 0:\n            return int(-c / b)\n        discriminant = b**2 - 4 * a * c\n        if discriminant < 0:\n            raise ValueError(f\"No real solution (discriminant = {discriminant})\")\n        root = (-b + sqrt(discriminant)) / (2 * a)\n        if root < 0:\n            raise ValueError(f\"No positive solution (root = {root})\")\n        return int(floor(root))\n","sourceCodeStart":716,"sourceCodeEnd":752,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/generation/continuous_batching/cache.py#L716-L752","documentation":"Thrown by PagedAttentionCache._check_footprint after the cache solves for max_batch_tokens/num_blocks: the memory polynomial evaluated at the chosen (max_batch_tokens, num_blocks) exceeds available_memory. It is a MemoryError raised during cache auto-sizing (get_max_batch_tokens_and_num_blocks), meaning the requested batch capacity physically cannot fit in the memory budget computed from the device.","triggerScenarios":"Calling the continuous-batching setup path that ends in _check_footprint (e.g. ContinuousBatchingManager creation) with an explicit max_batch_tokens or num_blocks whose footprint (activation peaks + KV cache) exceeds available_memory; or auto-sizing when available_memory was computed too optimistically (other tensors already allocated, small GPU).","commonSituations":"Small GPU (e.g. 8GB) with a large model or fp32 cache; leftover allocations from a previous run shrinking available_memory; user passes max_batch_tokens copied from a bigger-GPU setup; quantization/cache dtype not applied so cache bytes are 2x expected.","solutions":["Lower max_batch_tokens (or num_blocks) when constructing the ContinuousBatchingConfig / cache so the footprint fits available_memory","Reduce cache/activation memory: use a smaller cache dtype (e.g. float16/bfloat16 KV cache), a smaller model, or fewer layers","Free memory before manager creation: del previous managers/tensors and torch.cuda.empty_cache(), since available_memory is measured at setup time","If auto-sizing triggered it, report upstream — auto-sizing should never exceed the budget, so a rounding/ordering bug may exist in the solver"],"exampleFix":"# before\nconfig = ContinuousBatchingConfig(max_batch_tokens=32768)  # too large for the GPU\n\n# after\nconfig = ContinuousBatchingConfig(max_batch_tokens=4096)  # fit to available memory","handlingStrategy":"validation","validationCode":"free_b, total_b = torch.cuda.mem_get_info() if torch.cuda.is_available() else (psutil.virtual_memory().available, psutil.virtual_memory().total)\n# footprint grows with max_batch_tokens; start conservative and scale up\nassert free_b > 2 * 1024**3, f\"only {free_b/1024**3:.1f} GiB free — free memory before starting\"","typeGuard":null,"tryCatchPattern":"try:\n    manager = model.continuous_batching(config=cfg)\nexcept MemoryError as e:\n    cfg.max_batch_tokens = max(256, cfg.max_batch_tokens // 2)\n    torch.cuda.empty_cache()\n    manager = model.continuous_batching(config=cfg)","preventionTips":["Free GPU memory (del old managers, empty_cache) before creating the manager","Set cache dtype to fp16/bf16","Start with a small max_batch_tokens and benchmark upward"],"tags":["memory","continuous-batching","kv-cache","configuration"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}