apache/hadoop · error · RuntimeException

PipeMapRed.waitOutputThreads(): subprocess failed with code

Error message

PipeMapRed.waitOutputThreads(): subprocess failed with code {}

What it means

After the streaming subprocess (mapper/reducer executable) exits, PipeMapRed.waitOutputThreads() checks the exit code. With stream.non.zero.exit.is.failure=true (the default), any non-zero exit throws RuntimeException 'PipeMapRed.waitOutputThreads(): subprocess failed with code <N>'. The number is the external tool's own exit status, making this the primary diagnostic for crashed streaming executables.

Solutions

1. Rerun the executable standalone with representative input and fix whatever makes it exit non-zero (the exit code narrows it: 127=command not found, 126=not executable, 139=segfault)
2. Make scripts exit explicitly (end with 'exit 0') and catch exceptions so a few bad records don't abort the task
3. If non-zero exits are acceptable for your workflow, set -Dstream.non.zero.exit.is.failure=false so it is logged instead of failing the job
4. Increase tool robustness: validate input, add error handling around record parsing, and log tool stderr (it is captured in task logs)

Example fix

# before (my_mapper.py crashes on bad input and exits 1)
for line in sys.stdin:
    f = line.split('\t')
    print(f[0].upper(), f[1])
# after
import sys
for line in sys.stdin:
    f = line.rstrip('\n').split('\t')
    if len(f) < 2:
        sys.stderr.write('skipping bad record: %r\n' % line)
        continue
    print(f[0].upper(), f[1])

// preflight: run the tool on sample data and assert exit code 0 before submitting the job
Process p = new ProcessBuilder("./my_mapper.py").redirectInput(new File("sample.txt")).start();
int rc = p.waitFor();
if (rc != 0) throw new IllegalStateException("mapper exits " + rc + " on sample input; fix before submitting");

catch RuntimeException from job run; when the message matches 'subprocess failed with code N', decode N (127 command-not-found, 126 not-executable, 139 signal) and fix the tool rather than retrying blindly.

When it happens

Trigger: The external tool exits non-zero: an uncaught Python exception (exit 1), a shell script 'exit 3', a C program aborting, or the tool dying on a bad input record. With the default flag, the task fails; with stream.non.zero.exit.is.failure=false the exit is only logged at INFO.

Common situations: Streaming jobs where the mapper crashes on malformed records (quote/escape issues, missing fields), missing interpreters producing 127, tools killed by signals (138/139-style codes), or scripts that forget 'exit 0' and end with a failing last command (e.g., grep that finds nothing).

Related errors

• MROutput/MRErrThread failed:
• configuration exception
• Error executing command.
• MROutput/MRErrThread failed:
• A Mapper can be added to the chain only after the Reducer…

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f8706cb3f8ef8731. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/PipeMapRed.java:326

        // called at all in this task). If reducer still generates output,
        // which is very uncommon and we may not have to support this case.
        // So we don't write this output to HDFS, but we consume/collect
        // this output just to avoid reducer hanging forever.

        OutputCollector collector = new OutputCollector() {
          public void collect(Object key, Object value)
            throws IOException {
            //just consume it, no need to write the record anywhere
          }
        };
        Reporter reporter = Reporter.NULL;//dummy reporter
        startOutputThreads(collector, reporter);
      }
      int exitVal = sim.waitFor();
      // how'd it go?
      if (exitVal != 0) {
        if (nonZeroExitIsFailure_) {
          throw new RuntimeException("PipeMapRed.waitOutputThreads(): subprocess failed with code "
                                     + exitVal);
        } else {
          LOG.info("PipeMapRed.waitOutputThreads(): subprocess exited with " +
          		"code " + exitVal + " in " + PipeMapRed.class.getName());
        }
      }
      if (outThread_ != null) {
        outThread_.join(joinDelay_);
      }
      if (errThread_ != null) {
        errThread_.join(joinDelay_);
      }
      if (outerrThreadsThrowable != null) {
        throw new RuntimeException(outerrThreadsThrowable);
      }
    } catch (InterruptedException e) {
      //ignore
    }

View on GitHub (pinned to 2add963021)