{"record":{"id":"f8706cb3f8ef8731","repo":"apache/hadoop","slug":"pipemapred-waitoutputthreads-subprocess-failed","errorCode":null,"errorMessage":"PipeMapRed.waitOutputThreads(): subprocess failed with code {}","messagePattern":"PipeMapRed\\.waitOutputThreads\\(\\): subprocess failed with code (.+?)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/PipeMapRed.java","lineNumber":326,"sourceCode":"        // called at all in this task). If reducer still generates output,\n        // which is very uncommon and we may not have to support this case.\n        // So we don't write this output to HDFS, but we consume/collect\n        // this output just to avoid reducer hanging forever.\n\n        OutputCollector collector = new OutputCollector() {\n          public void collect(Object key, Object value)\n            throws IOException {\n            //just consume it, no need to write the record anywhere\n          }\n        };\n        Reporter reporter = Reporter.NULL;//dummy reporter\n        startOutputThreads(collector, reporter);\n      }\n      int exitVal = sim.waitFor();\n      // how'd it go?\n      if (exitVal != 0) {\n        if (nonZeroExitIsFailure_) {\n          throw new RuntimeException(\"PipeMapRed.waitOutputThreads(): subprocess failed with code \"\n                                     + exitVal);\n        } else {\n          LOG.info(\"PipeMapRed.waitOutputThreads(): subprocess exited with \" +\n          \t\t\"code \" + exitVal + \" in \" + PipeMapRed.class.getName());\n        }\n      }\n      if (outThread_ != null) {\n        outThread_.join(joinDelay_);\n      }\n      if (errThread_ != null) {\n        errThread_.join(joinDelay_);\n      }\n      if (outerrThreadsThrowable != null) {\n        throw new RuntimeException(outerrThreadsThrowable);\n      }\n    } catch (InterruptedException e) {\n      //ignore\n    }","sourceCodeStart":308,"sourceCodeEnd":344,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/PipeMapRed.java#L308-L344","documentation":"After the streaming subprocess (mapper/reducer executable) exits, PipeMapRed.waitOutputThreads() checks the exit code. With stream.non.zero.exit.is.failure=true (the default), any non-zero exit throws RuntimeException 'PipeMapRed.waitOutputThreads(): subprocess failed with code <N>'. The number is the external tool's own exit status, making this the primary diagnostic for crashed streaming executables.","triggerScenarios":"The external tool exits non-zero: an uncaught Python exception (exit 1), a shell script 'exit 3', a C program aborting, or the tool dying on a bad input record. With the default flag, the task fails; with stream.non.zero.exit.is.failure=false the exit is only logged at INFO.","commonSituations":"Streaming jobs where the mapper crashes on malformed records (quote/escape issues, missing fields), missing interpreters producing 127, tools killed by signals (138/139-style codes), or scripts that forget 'exit 0' and end with a failing last command (e.g., grep that finds nothing).","solutions":["Rerun the executable standalone with representative input and fix whatever makes it exit non-zero (the exit code narrows it: 127=command not found, 126=not executable, 139=segfault)","Make scripts exit explicitly (end with 'exit 0') and catch exceptions so a few bad records don't abort the task","If non-zero exits are acceptable for your workflow, set -Dstream.non.zero.exit.is.failure=false so it is logged instead of failing the job","Increase tool robustness: validate input, add error handling around record parsing, and log tool stderr (it is captured in task logs)"],"exampleFix":"# before (my_mapper.py crashes on bad input and exits 1)\nfor line in sys.stdin:\n    f = line.split('\\t')\n    print(f[0].upper(), f[1])\n# after\nimport sys\nfor line in sys.stdin:\n    f = line.rstrip('\\n').split('\\t')\n    if len(f) < 2:\n        sys.stderr.write('skipping bad record: %r\\n' % line)\n        continue\n    print(f[0].upper(), f[1])","handlingStrategy":"try-catch","validationCode":"// preflight: run the tool on sample data and assert exit code 0 before submitting the job\nProcess p = new ProcessBuilder(\"./my_mapper.py\").redirectInput(new File(\"sample.txt\")).start();\nint rc = p.waitFor();\nif (rc != 0) throw new IllegalStateException(\"mapper exits \" + rc + \" on sample input; fix before submitting\");","typeGuard":null,"tryCatchPattern":"catch RuntimeException from job run; when the message matches 'subprocess failed with code N', decode N (127 command-not-found, 126 not-executable, 139 signal) and fix the tool rather than retrying blindly.","preventionTips":["End shell scripts with explicit 'exit 0'; handle expected errors instead of letting them propagate","Print diagnostics to stderr; keep stdout for emitted records","Consider -Dstream.non.zero.exit.is.failure=false only when non-zero exits are genuinely acceptable","Include malformed-input handling in every streaming script"],"tags":["hadoop-streaming","subprocess","exit-code","scripting","mapreduce"],"backgroundTag":"subprocess-non-zero-exit","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}