docling-project/docling · error · ValueError

Total extracted data exceeds maximum limit of {self.options.

Error message

Total extracted data exceeds maximum limit of {self.options.max_total_bytes} bytes

What it means

ValueError raised after extracting a page image: adding that image's byte count pushes self._total_bytes_extracted above options.max_total_bytes. Unlike the init-time total (XML members only), this accumulator also counts image/OCR data read during page conversion, so big page scans can exhaust the global extraction budget mid-book.

Source

Thrown at docling/backend/mets_gbs_backend.py:402

        # Security: limit extraction size to prevent decompression bombs
        image_file = self._tar.extractfile(image_info.path)
        if image_file is None:
            raise RuntimeError(
                f"Archive member '{image_info.path}' is not a regular file "
                "(directory or symlink in tar)."
            )
        image_file = cast(tarfile.ExFileObject, image_file)
        image_data = image_file.read(self.options.max_file_bytes + 1)
        if len(image_data) > self.options.max_file_bytes:
            raise ValueError(
                f"Image file {image_info.path} exceeds individual file size limit of {self.options.max_file_bytes} bytes"
            )

        # Security: Track total bytes extracted
        self._total_bytes_extracted += len(image_data)
        if self._total_bytes_extracted > self.options.max_total_bytes:
            raise ValueError(
                f"Total extracted data exceeds maximum limit of {self.options.max_total_bytes} bytes"
            )

        buf = BytesIO(image_data)
        im: PILImage = Image.open(buf)

        ocr_file = self._tar.extractfile(ocr_info.path)
        if ocr_file is None:
            raise RuntimeError(
                f"Archive member '{ocr_info.path}' is not a regular file "
                "(directory or symlink in tar)."
            )
        ocr_file = cast(tarfile.ExFileObject, ocr_file)
        ocr_content = ocr_file.read(self.options.max_file_bytes + 1)
        if len(ocr_content) > self.options.max_file_bytes:
            raise ValueError(
                f"OCR file {ocr_info.path} exceeds individual file size limit of {self.options.max_file_bytes} bytes"
            )

View on GitHub (pinned to 61d76f1ff3)

Solutions

Increase MetsGbsBackendOptions(max_total_bytes=...) to cover all pages' combined bytes.
Reduce per-image size (downsample scans) so the total stays under budget.
Split multi-volume archives and convert each volume separately.
Monitor memory/disk if you raise the limit substantially — the cap also bounds peak resource use.

Example fix

# before
result = converter.convert(mets_path)  # ValueError mid-book: total extracted

# after
opts = MetsGbsBackendOptions(max_total_bytes=10 * 1024 * 1024 * 1024)
# wire into converter format options, then convert
result = converter.convert(mets_path)

Defensive patterns

Strategy: validation

Validate before calling

import tarfile

def total_uncompressed(tar_path: str) -> int:
    with tarfile.open(tar_path) as t:
        return sum(m.size for m in t.getmembers())

# pass MetsGbsBackendOptions(max_total_bytes=int(total_uncompressed(p) * 1.2))

Try / catch

try:
    result = converter.convert(mets_path)
except ValueError as e:
    if 'maximum limit' in str(e) or 'Total extracted data' in str(e):
        log.error('archive %s exceeds extraction budget — split or raise max_total_bytes', mets_path)

Prevention

Compute total uncompressed size from tar headers before conversion and budget max_total_bytes above it.
Remember images and OCR add to the same accumulator as XML.
Split multi-volume books so each archive fits the budget.

When it happens

Trigger: Converting a many-page METS book where cumulative extracted bytes (XML + images + OCR across pages processed so far) exceed max_total_bytes; typically trips partway through conversion on a later page.

Common situations: Long books with high-resolution scans, batch conversion reusing one options object with an aggressive total cap, or archives combining bulky XML plus bulky images.

Related errors

AI-assisted analysis of docling-project/docling@61d76f1ff3 (2026-08-14). Data as JSON: /api/errors/3ac338fc2ea08610. Report an issue: GitHub.