apache/hadoop · error · RuntimeException

PipeMapRed.waitOutputThreads(): subprocess failed with code

Error message

PipeMapRed.waitOutputThreads(): subprocess failed with code {}

What it means

After the streaming subprocess (mapper/reducer executable) exits, PipeMapRed.waitOutputThreads() checks the exit code. With stream.non.zero.exit.is.failure=true (the default), any non-zero exit throws RuntimeException 'PipeMapRed.waitOutputThreads(): subprocess failed with code <N>'. The number is the external tool's own exit status, making this the primary diagnostic for crashed streaming executables.

Source

Thrown at hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/PipeMapRed.java:326

        // called at all in this task). If reducer still generates output,
        // which is very uncommon and we may not have to support this case.
        // So we don't write this output to HDFS, but we consume/collect
        // this output just to avoid reducer hanging forever.

        OutputCollector collector = new OutputCollector() {
          public void collect(Object key, Object value)
            throws IOException {
            //just consume it, no need to write the record anywhere
          }
        };
        Reporter reporter = Reporter.NULL;//dummy reporter
        startOutputThreads(collector, reporter);
      }
      int exitVal = sim.waitFor();
      // how'd it go?
      if (exitVal != 0) {
        if (nonZeroExitIsFailure_) {
          throw new RuntimeException("PipeMapRed.waitOutputThreads(): subprocess failed with code "
                                     + exitVal);
        } else {
          LOG.info("PipeMapRed.waitOutputThreads(): subprocess exited with " +
          		"code " + exitVal + " in " + PipeMapRed.class.getName());
        }
      }
      if (outThread_ != null) {
        outThread_.join(joinDelay_);
      }
      if (errThread_ != null) {
        errThread_.join(joinDelay_);
      }
      if (outerrThreadsThrowable != null) {
        throw new RuntimeException(outerrThreadsThrowable);
      }
    } catch (InterruptedException e) {
      //ignore
    }

View on GitHub (pinned to 2add963021)

Solutions

Rerun the executable standalone with representative input and fix whatever makes it exit non-zero (the exit code narrows it: 127=command not found, 126=not executable, 139=segfault)
Make scripts exit explicitly (end with 'exit 0') and catch exceptions so a few bad records don't abort the task
If non-zero exits are acceptable for your workflow, set -Dstream.non.zero.exit.is.failure=false so it is logged instead of failing the job
Increase tool robustness: validate input, add error handling around record parsing, and log tool stderr (it is captured in task logs)

Example fix

# before (my_mapper.py crashes on bad input and exits 1)
for line in sys.stdin:
    f = line.split('\t')
    print(f[0].upper(), f[1])
# after
import sys
for line in sys.stdin:
    f = line.rstrip('\n').split('\t')
    if len(f) < 2:
        sys.stderr.write('skipping bad record: %r\n' % line)
        continue
    print(f[0].upper(), f[1])

Defensive patterns

Strategy: try-catch

Validate before calling

// preflight: run the tool on sample data and assert exit code 0 before submitting the job
Process p = new ProcessBuilder("./my_mapper.py").redirectInput(new File("sample.txt")).start();
int rc = p.waitFor();
if (rc != 0) throw new IllegalStateException("mapper exits " + rc + " on sample input; fix before submitting");

Try / catch

catch RuntimeException from job run; when the message matches 'subprocess failed with code N', decode N (127 command-not-found, 126 not-executable, 139 signal) and fix the tool rather than retrying blindly.

Prevention

End shell scripts with explicit 'exit 0'; handle expected errors instead of letting them propagate
Print diagnostics to stderr; keep stdout for emitted records
Consider -Dstream.non.zero.exit.is.failure=false only when non-zero exits are genuinely acceptable
Include malformed-input handling in every streaming script

When it happens

Trigger: The external tool exits non-zero: an uncaught Python exception (exit 1), a shell script 'exit 3', a C program aborting, or the tool dying on a bad input record. With the default flag, the task fails; with stream.non.zero.exit.is.failure=false the exit is only logged at INFO.

Common situations: Streaming jobs where the mapper crashes on malformed records (quote/escape issues, missing fields), missing interpreters producing 127, tools killed by signals (138/139-style codes), or scripts that forget 'exit 0' and end with a failing last command (e.g., grep that finds nothing).

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f8706cb3f8ef8731. Report an issue: GitHub.