apache/hadoop · error · IOException

MROutput/MRErrThread failed:

Error message

MROutput/MRErrThread failed:

What it means

PipeMapper.map() checks the shared field outerrThreadsThrowable before processing each record: if the MROutputThread (parsing the tool's stdout) or MRErrThread (draining stderr) has died, it calls mapRedFinished() and throws IOException('MROutput/MRErrThread failed:', outerrThreadsThrowable) — the real failure is the wrapped cause, not this message.

Source

Thrown at hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/PipeMapper.java:92

      String inputFormatClassName = job.getClass("mapred.input.format.class", TextInputFormat.class).getCanonicalName();
      ignoreKey = job.getBoolean("stream.map.input.ignoreKey", 
        inputFormatClassName.equals(TextInputFormat.class.getCanonicalName()));
    }
    
    mapOutputFieldSeparator = job.get("stream.map.output.field.separator", "\t")
            .getBytes(StandardCharsets.UTF_8);
    mapInputFieldSeparator = job.get("stream.map.input.field.separator", "\t")
            .getBytes(StandardCharsets.UTF_8);
    numOfMapOutputKeyFields = job.getInt("stream.num.map.output.key.fields", 1);
  }

  // Do NOT declare default constructor
  // (MapRed creates it reflectively)

  public void map(Object key, Object value, OutputCollector output, Reporter reporter) throws IOException {
    if (outerrThreadsThrowable != null) {
      mapRedFinished();
      throw new IOException("MROutput/MRErrThread failed:",
          outerrThreadsThrowable);
    }
    try {
      // 1/4 Hadoop in
      numRecRead_++;
      maybeLogRecord();

      // 2/4 Hadoop to Tool
      if (numExceptions_ == 0) {
        if (!this.ignoreKey) {
          inWriter_.writeKey(key);
        }
        inWriter_.writeValue(value);
        if(skipping) {
          //flush the streams on every record input if running in skip mode
          //so that we don't buffer other records surrounding a bad record. 
          clientOut_.flush();
        }

View on GitHub (pinned to 2add963021)

Solutions

1. Look earlier in the task log for the wrapped throwable from MROutputThread/MRErrThread — that stack trace, not this IOException, is the root cause
2. Make the tool's stdout format match the configured reader and key-field settings (or use a custom OutputReader)
3. Send diagnostics to stderr only; keep stdout strictly to emitted key\tvalue records
4. Fix the underlying tool crash (often pairs with a non-zero exit — see the subprocess failed-with-code error)

// preflight: ensure tool stdout matches what the OutputReader expects before job submit
// e.g. for the default reader: exactly one separator, correct key-field count
try (BufferedReader r = new BufferedReader(new FileReader("tool-sample-output.txt"))) {
  String line; int bad = 0;
  while ((line = r.readLine()) != null) {
    int tabs = line.length() - line.replace("\t", "").length();
    if (tabs < 1) bad++;
  }
  if (bad > 0) throw new IllegalStateException(bad + " output lines lack a tab separator");
}

catch IOException from map()/the streaming job; drill into getCause() (the MROutputThread/MRErrThread throwable) — the wrapper message alone says nothing about the real fault.

When it happens

Trigger: An output-reader thread threw earlier: malformed tool output that the OutputReader cannot parse (custom stream.map.output.reader.class), broken pipe after the tool exited, or exceptions inside the stderr-draining thread; the very next map() call then aborts with this wrapper.

Common situations: Tools printing non-key-value output when a parser expects fields (stream.num.map.output.key.fields mismatches), tools writing diagnostics to stdout instead of stderr, or the tool crashing mid-stream so the reader hits EOF/pipe errors; the task log's earlier 'MROutputThread/MRErrThread' stack trace is the actual root cause.

Related errors

• MROutput/MRErrThread failed:
• PipeMapRed.waitOutputThreads(): subprocess failed with code
• configuration exception
• Unable to recover task %s, output: %s
• Error executing command. %s Process exited with exit code %d

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/46c6c5ff91e5b883. Report an issue: GitHub.