opendataloader-project/opendataloader-pdf · error · IOException

Parallel page processing failed

Error message

Parallel page processing failed

What it means

A catch-all IOException wrapping ANY exception thrown while the ForkJoinPool processes pages in parallel (ParagraphProcessor, ListProcessor, HeadingProcessor) or during the sequential cross-page post-processing that follows (checkNeighborLists, checkNeighborTables, detectHeadingsLevels, detectLevels). The original cause is preserved via IOException#getCause. Because the pool runs IntStream.range().parallel(), an uncaught exception on a worker thread surfaces here as an ExecutionException unwrapped by .get().

Source

Thrown at java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/processors/DocumentProcessor.java:430

            // Caption detection runs after setIDs so that recognizedStructureId is available
            // for linking captions to figures/tables
            if (structured) {
                for (int pageNumber = 0; pageNumber < totalPages; pageNumber++) {
                    if (shouldProcessPage(pageNumber, pagesToProcess)) {
                        CaptionProcessor.processCaptions(contents.get(pageNumber));
                    }
                }
            }

            if (structured) {
                // Cross-page post-processing (must be sequential)
                ListProcessor.checkNeighborLists(contents);
                TableBorderProcessor.checkNeighborTables(contents);
                HeadingProcessor.detectHeadingsLevels();
                LevelProcessor.detectLevels(contents);
            }
        } catch (Exception e) {
            throw new IOException("Parallel page processing failed", e);
        } finally {
            pool.shutdown();
        }
        return contents;
    }

    /**
     * Checks if a page should be processed based on the filter.
     *
     * @param pageNumber 0-indexed page number
     * @param pagesToProcess set of valid page numbers to process, or null for all pages
     * @return true if the page should be processed
     */
    /**
     * Filters ElementMetadata down to entries whose transformer-assigned ID still
     * matches an IObject in the post-enrichment contents. This is deliberately
     * ID-based (not positional): sorting, filtering, and enrichment can reorder
     * or drop IObjects, so positional matching would attach the wrong

View on GitHub (pinned to a7789b8e77)

Solutions

1. Read the wrapped cause: catch the IOException and log/print getCause().getStackTrace() — the real failing processor and line are there, not in this wrapper.
2. If the cause is an NPE on a StaticContainers/StaticLayoutContainers field, find the processor that reads it and ensure that ThreadLocal is set inside the propagateState Runnable passed to each worker (see the CLAUDE.md ThreadLocal propagation note).
3. Reproduce single-threaded by temporarily forcing the pool to parallelism 1 or processing only the offending page (--pages) to isolate which page/processor fails.
4. If the cause originates in a cross-page sequential processor (after the parallel block), the failure is deterministic per document — re-run with that document and debug the named processor directly.

Example fix

// before: a new processor reads a ThreadLocal not propagated to workers
Runnable propagateState = () -> {
    StaticContainers.setDocument(document);
    // StaticLayoutContainers.setLayoutConfig(...) MISSING -> NPE on worker
};
// after: include every ThreadLocal the new processor reads
Runnable propagateState = () -> {
    StaticContainers.setDocument(document);
    StaticLayoutContainers.setLayoutConfig(config.getLayoutConfig());
};

// No pre-check possible for arbitrary processing failures; validate inputs earlier.
// You can bound the blast radius by processing a single page first to surface worker errors:
DocumentProcessor.extractContents(singlePagePdf, config); // smoke test before batch

// Java has no type guard; narrow on exception type and cause.
static boolean isParallelProcessingFailure(IOException e) {
    return "Parallel page processing failed".equals(e.getMessage());
}

try {
    contents = DocumentProcessor.extractContents(pdfName, config);
} catch (IOException e) {
    if (e.getCause() != null && e.getCause() instanceof NullPointerException) {
        // Likely a ThreadLocal propagation bug in a custom processor/extension
        log.error("Worker NPE — check propagateState covers all StaticContainers used", e.getCause());
    } else {
        log.error("Processing failed: {}", e.getMessage(), e.getCause());
    }
    throw e;
}

When it happens

Trigger: A worker thread throws during per-page processing (NPE from a missing ThreadLocal because propagateState.run() was not called for a new processor, ArrayIndexOutOfBounds on malformed page content), or a sequential cross-page processor (ListProcessor.checkNeighborLists, TableBorderProcessor.checkNeighborTables, HeadingProcessor.detectHeadingsLevels, LevelProcessor.detectLevels) throws on the main thread inside the same try block.

Common situations: A new processor added to the parallel block reads a StaticContainers/StaticLayoutContainers ThreadLocal that was never registered in propagateState, causing silent NPEs only under parallel mode. Corrupt page objects that survive parsing but break a processor. Regressions after refactoring shared static state.

AI-assisted analysis of opendataloader-project/opendataloader-pdf@a7789b8e77 (2026-08-14). Data as JSON: /api/errors/c2d4880f5d07eb78. Report an issue: GitHub.