{"record":{"id":"46c6c5ff91e5b883","repo":"apache/hadoop","slug":"mroutput-mrerrthread-failed","errorCode":null,"errorMessage":"MROutput/MRErrThread failed:","messagePattern":"MROutput/MRErrThread failed:","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/PipeMapper.java","lineNumber":92,"sourceCode":"      String inputFormatClassName = job.getClass(\"mapred.input.format.class\", TextInputFormat.class).getCanonicalName();\n      ignoreKey = job.getBoolean(\"stream.map.input.ignoreKey\", \n        inputFormatClassName.equals(TextInputFormat.class.getCanonicalName()));\n    }\n    \n    mapOutputFieldSeparator = job.get(\"stream.map.output.field.separator\", \"\\t\")\n            .getBytes(StandardCharsets.UTF_8);\n    mapInputFieldSeparator = job.get(\"stream.map.input.field.separator\", \"\\t\")\n            .getBytes(StandardCharsets.UTF_8);\n    numOfMapOutputKeyFields = job.getInt(\"stream.num.map.output.key.fields\", 1);\n  }\n\n  // Do NOT declare default constructor\n  // (MapRed creates it reflectively)\n\n  public void map(Object key, Object value, OutputCollector output, Reporter reporter) throws IOException {\n    if (outerrThreadsThrowable != null) {\n      mapRedFinished();\n      throw new IOException(\"MROutput/MRErrThread failed:\",\n          outerrThreadsThrowable);\n    }\n    try {\n      // 1/4 Hadoop in\n      numRecRead_++;\n      maybeLogRecord();\n\n      // 2/4 Hadoop to Tool\n      if (numExceptions_ == 0) {\n        if (!this.ignoreKey) {\n          inWriter_.writeKey(key);\n        }\n        inWriter_.writeValue(value);\n        if(skipping) {\n          //flush the streams on every record input if running in skip mode\n          //so that we don't buffer other records surrounding a bad record. \n          clientOut_.flush();\n        }","sourceCodeStart":74,"sourceCodeEnd":110,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/PipeMapper.java#L74-L110","documentation":"PipeMapper.map() checks the shared field outerrThreadsThrowable before processing each record: if the MROutputThread (parsing the tool's stdout) or MRErrThread (draining stderr) has died, it calls mapRedFinished() and throws IOException('MROutput/MRErrThread failed:', outerrThreadsThrowable) — the real failure is the wrapped cause, not this message.","triggerScenarios":"An output-reader thread threw earlier: malformed tool output that the OutputReader cannot parse (custom stream.map.output.reader.class), broken pipe after the tool exited, or exceptions inside the stderr-draining thread; the very next map() call then aborts with this wrapper.","commonSituations":"Tools printing non-key-value output when a parser expects fields (stream.num.map.output.key.fields mismatches), tools writing diagnostics to stdout instead of stderr, or the tool crashing mid-stream so the reader hits EOF/pipe errors; the task log's earlier 'MROutputThread/MRErrThread' stack trace is the actual root cause.","solutions":["Look earlier in the task log for the wrapped throwable from MROutputThread/MRErrThread — that stack trace, not this IOException, is the root cause","Make the tool's stdout format match the configured reader and key-field settings (or use a custom OutputReader)","Send diagnostics to stderr only; keep stdout strictly to emitted key\\tvalue records","Fix the underlying tool crash (often pairs with a non-zero exit — see the subprocess failed-with-code error)"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// preflight: ensure tool stdout matches what the OutputReader expects before job submit\n// e.g. for the default reader: exactly one separator, correct key-field count\ntry (BufferedReader r = new BufferedReader(new FileReader(\"tool-sample-output.txt\"))) {\n  String line; int bad = 0;\n  while ((line = r.readLine()) != null) {\n    int tabs = line.length() - line.replace(\"\\t\", \"\").length();\n    if (tabs < 1) bad++;\n  }\n  if (bad > 0) throw new IllegalStateException(bad + \" output lines lack a tab separator\");\n}","typeGuard":null,"tryCatchPattern":"catch IOException from map()/the streaming job; drill into getCause() (the MROutputThread/MRErrThread throwable) — the wrapper message alone says nothing about the real fault.","preventionTips":["Write only key\\tvalue lines to stdout; send logs to stderr","Match stream.num.map.output.key.fields to the tool's actual output","Test tools locally: cat input | tool | head, verify format and exit code","Read the task log around the failure for the reader thread's own stack trace"],"tags":["hadoop-streaming","output-parsing","threads","mapreduce","subprocess"],"backgroundTag":"worker-thread-failure","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}