{"record":{"id":"08f823cf44610905","repo":"apache/seatunnel","slug":"failed-to-consume-python-source-stderr","errorCode":null,"errorMessage":"Failed to consume python source stderr","messagePattern":"Failed to consume python source stderr","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"critical","filePath":"seatunnel-connectors-v2/connector-python/src/main/java/org/apache/seatunnel/connectors/seatunnel/python/source/PythonSourceReader.java","lineNumber":629,"sourceCode":"            return;\n        }\n        try {\n            thread.join(PROCESS_EXIT_CHECK_TIMEOUT_MILLIS);\n        } catch (InterruptedException e) {\n            Thread.currentThread().interrupt();\n            throw new IOException(\"Interrupted while draining python source process output\", e);\n        }\n    }\n\n    private void checkPumpFailures() {\n        if (stdoutPumpFailure != null) {\n            throw new IllegalStateException(\n                    \"Failed to consume python source stdout\", stdoutPumpFailure);\n        }\n        if (stderrPumpFailure == null) {\n            return;\n        }\n        throw new IllegalStateException(\n                \"Failed to consume python source stderr\", stderrPumpFailure);\n    }\n\n    /** Registers one poll so close cannot return while rows or completion are still emitted. */\n    private boolean beginPoll() {\n        synchronized (lifecycleLock) {\n            if (noMoreSplits || closeRequested) {\n                return false;\n            }\n            activePolls++;\n            return true;\n        }\n    }\n\n    /** Releases the close barrier after every row and terminal signal has completed. */\n    private void endPoll() {\n        synchronized (lifecycleLock) {\n            activePolls--;","sourceCodeStart":611,"sourceCodeEnd":647,"githubUrl":"https://github.com/apache/seatunnel/blob/cf67b549a7a6c35fa0beb12d83c62892427ea919/seatunnel-connectors-v2/connector-python/src/main/java/org/apache/seatunnel/connectors/seatunnel/python/source/PythonSourceReader.java#L611-L647","documentation":"PythonSourceReader runs a background thread consuming the Python subprocess's stderr to prevent pipe deadlock. If that stderr pump thread throws, the failure is stored in stderrPumpFailure and rethrown as an IllegalStateException by checkPumpFailures, which is invoked from pollNext and verifyProcessExit. It signals that diagnostics from the Python process can no longer be captured and the reader is in a failed state.","triggerScenarios":"The stderr pump thread throws while reading the child's stderr stream; the failure is rethrown on the next pollNext or verifyProcessExit call through checkPumpFailures.","commonSituations":"Python process dies abruptly (OOM-kill, SIGKILL) closing the stderr pipe; I/O error reading the pipe; overly verbose stderr causing backpressure or reader errors; environment issues causing the Python interpreter to emit unexpected output.","solutions":["Inspect getCause() for the underlying stderr pump failure","Run the Python script manually and watch stderr for crash messages (tracebacks, OOM, import errors)","Ensure the child process is not being killed by the OS (check dmesg/oom-killer logs, raise memory limits)","Confirm Python interpreter version and dependencies are correct on every worker node"],"exampleFix":"// before\nthrow new IllegalStateException(\n        \"Failed to consume python source stderr\", stderrPumpFailure);\n// after\nLOG.error(\"python stderr pump failed; inspect python script stderr for a traceback\",\n        stderrPumpFailure.getCause());\nthrow new IllegalStateException(\n        \"Failed to consume python source stderr\", stderrPumpFailure);","handlingStrategy":"try-catch","validationCode":"// run the script and capture stderr before deploying\nProcess p = new ProcessBuilder(pythonScriptArgs).redirectErrorStream(false).start();\nString stderr = new String(p.getErrorStream().readAllBytes());\nif (!stderr.isEmpty()) LOG.warn(\"python stderr during dry-run: {}\", stderr);","typeGuard":null,"tryCatchPattern":"try {\n    reader.pollNext();\n} catch (IllegalStateException e) {\n    if (e.getMessage().contains(\"Failed to consume python source stderr\")) {\n        LOG.error(\"python stderr pump failed\", e.getCause());\n    } else throw e;\n}","preventionTips":["Keep python stderr output reasonable; avoid massive tracebacks flooding the pipe","Check the child process is not killed abruptly (memory limits, container OOM)","Validate python environment (interpreter, packages) on every worker node","Monitor job logs for earlier stderr content indicating the script was failing"],"tags":["python","subprocess","stream-consumption","process-crash"],"backgroundTag":"broken-pipe","analyzedSha":"cf67b549a7a6c35fa0beb12d83c62892427ea919","analyzedAt":"2026-09-10T21:44:55.265Z","contentChangedAt":"2026-09-10T21:44:55.265Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}