{"record":{"id":"c2d4880f5d07eb78","repo":"opendataloader-project/opendataloader-pdf","slug":"parallel-page-processing-failed","errorCode":null,"errorMessage":"Parallel page processing failed","messagePattern":"Parallel page processing failed","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/processors/DocumentProcessor.java","lineNumber":430,"sourceCode":"            // Caption detection runs after setIDs so that recognizedStructureId is available\n            // for linking captions to figures/tables\n            if (structured) {\n                for (int pageNumber = 0; pageNumber < totalPages; pageNumber++) {\n                    if (shouldProcessPage(pageNumber, pagesToProcess)) {\n                        CaptionProcessor.processCaptions(contents.get(pageNumber));\n                    }\n                }\n            }\n\n            if (structured) {\n                // Cross-page post-processing (must be sequential)\n                ListProcessor.checkNeighborLists(contents);\n                TableBorderProcessor.checkNeighborTables(contents);\n                HeadingProcessor.detectHeadingsLevels();\n                LevelProcessor.detectLevels(contents);\n            }\n        } catch (Exception e) {\n            throw new IOException(\"Parallel page processing failed\", e);\n        } finally {\n            pool.shutdown();\n        }\n        return contents;\n    }\n\n    /**\n     * Checks if a page should be processed based on the filter.\n     *\n     * @param pageNumber 0-indexed page number\n     * @param pagesToProcess set of valid page numbers to process, or null for all pages\n     * @return true if the page should be processed\n     */\n    /**\n     * Filters ElementMetadata down to entries whose transformer-assigned ID still\n     * matches an IObject in the post-enrichment contents. This is deliberately\n     * ID-based (not positional): sorting, filtering, and enrichment can reorder\n     * or drop IObjects, so positional matching would attach the wrong","sourceCodeStart":412,"sourceCodeEnd":448,"githubUrl":"https://github.com/opendataloader-project/opendataloader-pdf/blob/a7789b8e77dd05e2b8659eb3ea12fc458f80bfb8/java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/processors/DocumentProcessor.java#L412-L448","documentation":"A catch-all IOException wrapping ANY exception thrown while the ForkJoinPool processes pages in parallel (ParagraphProcessor, ListProcessor, HeadingProcessor) or during the sequential cross-page post-processing that follows (checkNeighborLists, checkNeighborTables, detectHeadingsLevels, detectLevels). The original cause is preserved via IOException#getCause. Because the pool runs IntStream.range().parallel(), an uncaught exception on a worker thread surfaces here as an ExecutionException unwrapped by .get().","triggerScenarios":"A worker thread throws during per-page processing (NPE from a missing ThreadLocal because propagateState.run() was not called for a new processor, ArrayIndexOutOfBounds on malformed page content), or a sequential cross-page processor (ListProcessor.checkNeighborLists, TableBorderProcessor.checkNeighborTables, HeadingProcessor.detectHeadingsLevels, LevelProcessor.detectLevels) throws on the main thread inside the same try block.","commonSituations":"A new processor added to the parallel block reads a StaticContainers/StaticLayoutContainers ThreadLocal that was never registered in propagateState, causing silent NPEs only under parallel mode. Corrupt page objects that survive parsing but break a processor. Regressions after refactoring shared static state.","solutions":["Read the wrapped cause: catch the IOException and log/print getCause().getStackTrace() — the real failing processor and line are there, not in this wrapper.","If the cause is an NPE on a StaticContainers/StaticLayoutContainers field, find the processor that reads it and ensure that ThreadLocal is set inside the propagateState Runnable passed to each worker (see the CLAUDE.md ThreadLocal propagation note).","Reproduce single-threaded by temporarily forcing the pool to parallelism 1 or processing only the offending page (--pages) to isolate which page/processor fails.","If the cause originates in a cross-page sequential processor (after the parallel block), the failure is deterministic per document — re-run with that document and debug the named processor directly."],"exampleFix":"// before: a new processor reads a ThreadLocal not propagated to workers\nRunnable propagateState = () -> {\n    StaticContainers.setDocument(document);\n    // StaticLayoutContainers.setLayoutConfig(...) MISSING -> NPE on worker\n};\n// after: include every ThreadLocal the new processor reads\nRunnable propagateState = () -> {\n    StaticContainers.setDocument(document);\n    StaticLayoutContainers.setLayoutConfig(config.getLayoutConfig());\n};","handlingStrategy":"try-catch","validationCode":"// No pre-check possible for arbitrary processing failures; validate inputs earlier.\n// You can bound the blast radius by processing a single page first to surface worker errors:\nDocumentProcessor.extractContents(singlePagePdf, config); // smoke test before batch","typeGuard":"// Java has no type guard; narrow on exception type and cause.\nstatic boolean isParallelProcessingFailure(IOException e) {\n    return \"Parallel page processing failed\".equals(e.getMessage());\n}","tryCatchPattern":"try {\n    contents = DocumentProcessor.extractContents(pdfName, config);\n} catch (IOException e) {\n    if (e.getCause() != null && e.getCause() instanceof NullPointerException) {\n        // Likely a ThreadLocal propagation bug in a custom processor/extension\n        log.error(\"Worker NPE — check propagateState covers all StaticContainers used\", e.getCause());\n    } else {\n        log.error(\"Processing failed: {}\", e.getMessage(), e.getCause());\n    }\n    throw e;\n}","preventionTips":["When adding a processor to the parallel block, add every StaticContainers/StaticLayoutContainers ThreadLocal it reads to the propagateState Runnable (see CLAUDE.md).","Test new processors single-threaded (parallelism 1) and on a small PDF before enabling full ForkJoinPool parallelism.","Never swallow the cause — always log getCause() so the real failing line is visible."],"tags":["parallel","forkjoinpool","threadlocal","pdf-processing","wrapper-exception"],"backgroundTag":null,"analyzedSha":"a7789b8e77dd05e2b8659eb3ea12fc458f80bfb8","analyzedAt":"2026-08-14T05:22:03.953Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}