{"record":{"id":"3ac338fc2ea08610","repo":"docling-project/docling","slug":"total-extracted-data-exceeds-maximum-limit-of-sel","errorCode":null,"errorMessage":"Total extracted data exceeds maximum limit of {self.options.max_total_bytes} bytes","messagePattern":"Total extracted data exceeds maximum limit of (.+?) bytes","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"docling/backend/mets_gbs_backend.py","lineNumber":402,"sourceCode":"\n        # Security: limit extraction size to prevent decompression bombs\n        image_file = self._tar.extractfile(image_info.path)\n        if image_file is None:\n            raise RuntimeError(\n                f\"Archive member '{image_info.path}' is not a regular file \"\n                \"(directory or symlink in tar).\"\n            )\n        image_file = cast(tarfile.ExFileObject, image_file)\n        image_data = image_file.read(self.options.max_file_bytes + 1)\n        if len(image_data) > self.options.max_file_bytes:\n            raise ValueError(\n                f\"Image file {image_info.path} exceeds individual file size limit of {self.options.max_file_bytes} bytes\"\n            )\n\n        # Security: Track total bytes extracted\n        self._total_bytes_extracted += len(image_data)\n        if self._total_bytes_extracted > self.options.max_total_bytes:\n            raise ValueError(\n                f\"Total extracted data exceeds maximum limit of {self.options.max_total_bytes} bytes\"\n            )\n\n        buf = BytesIO(image_data)\n        im: PILImage = Image.open(buf)\n\n        ocr_file = self._tar.extractfile(ocr_info.path)\n        if ocr_file is None:\n            raise RuntimeError(\n                f\"Archive member '{ocr_info.path}' is not a regular file \"\n                \"(directory or symlink in tar).\"\n            )\n        ocr_file = cast(tarfile.ExFileObject, ocr_file)\n        ocr_content = ocr_file.read(self.options.max_file_bytes + 1)\n        if len(ocr_content) > self.options.max_file_bytes:\n            raise ValueError(\n                f\"OCR file {ocr_info.path} exceeds individual file size limit of {self.options.max_file_bytes} bytes\"\n            )","sourceCodeStart":384,"sourceCodeEnd":420,"githubUrl":"https://github.com/docling-project/docling/blob/61d76f1ff3f8428065465889f7b4577da7df704c/docling/backend/mets_gbs_backend.py#L384-L420","documentation":"ValueError raised after extracting a page image: adding that image's byte count pushes self._total_bytes_extracted above options.max_total_bytes. Unlike the init-time total (XML members only), this accumulator also counts image/OCR data read during page conversion, so big page scans can exhaust the global extraction budget mid-book.","triggerScenarios":"Converting a many-page METS book where cumulative extracted bytes (XML + images + OCR across pages processed so far) exceed max_total_bytes; typically trips partway through conversion on a later page.","commonSituations":"Long books with high-resolution scans, batch conversion reusing one options object with an aggressive total cap, or archives combining bulky XML plus bulky images.","solutions":["Increase MetsGbsBackendOptions(max_total_bytes=...) to cover all pages' combined bytes.","Reduce per-image size (downsample scans) so the total stays under budget.","Split multi-volume archives and convert each volume separately.","Monitor memory/disk if you raise the limit substantially — the cap also bounds peak resource use."],"exampleFix":"# before\nresult = converter.convert(mets_path)  # ValueError mid-book: total extracted\n\n# after\nopts = MetsGbsBackendOptions(max_total_bytes=10 * 1024 * 1024 * 1024)\n# wire into converter format options, then convert\nresult = converter.convert(mets_path)","handlingStrategy":"validation","validationCode":"import tarfile\n\ndef total_uncompressed(tar_path: str) -> int:\n    with tarfile.open(tar_path) as t:\n        return sum(m.size for m in t.getmembers())\n\n# pass MetsGbsBackendOptions(max_total_bytes=int(total_uncompressed(p) * 1.2))","typeGuard":null,"tryCatchPattern":"try:\n    result = converter.convert(mets_path)\nexcept ValueError as e:\n    if 'maximum limit' in str(e) or 'Total extracted data' in str(e):\n        log.error('archive %s exceeds extraction budget — split or raise max_total_bytes', mets_path)","preventionTips":["Compute total uncompressed size from tar headers before conversion and budget max_total_bytes above it.","Remember images and OCR add to the same accumulator as XML.","Split multi-volume books so each archive fits the budget."],"tags":["mets","archive","security","limits","decompression-bomb"],"backgroundTag":null,"analyzedSha":"61d76f1ff3f8428065465889f7b4577da7df704c","analyzedAt":"2026-08-14T23:53:18.727Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}