{"record":{"id":"d96f037affb70678","repo":"apache/hadoop","slug":"configuration-exception","errorCode":null,"errorMessage":"configuration exception","messagePattern":"configuration exception","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"critical","filePath":"hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/PipeMapRed.java","lineNumber":222,"sourceCode":"      envPut(childEnv, \"TMPDIR\", System.getProperty(\"java.io.tmpdir\"));\n\n      // Start the process\n      ProcessBuilder builder = new ProcessBuilder(argvSplit);\n      builder.environment().putAll(childEnv.toMap());\n      sim = builder.start();\n\n      clientOut_ = new DataOutputStream(new BufferedOutputStream(\n                                              sim.getOutputStream(),\n                                              BUFFER_SIZE));\n      clientIn_ = new DataInputStream(new BufferedInputStream(\n                                              sim.getInputStream(),\n                                              BUFFER_SIZE));\n      clientErr_ = new DataInputStream(new BufferedInputStream(sim.getErrorStream()));\n      startTime_ = System.currentTimeMillis();\n\n    } catch (IOException e) {\n      LOG.error(\"configuration exception\", e);\n      throw new RuntimeException(\"configuration exception\", e);\n    } catch (InterruptedException e)  {\n      LOG.error(\"configuration exception\", e);\n      throw new RuntimeException(\"configuration exception\", e);\n    }\n  }\n  \n  void setStreamJobDetails(JobConf job) {\n    String s = job.get(\"stream.minRecWrittenToEnableSkip_\");\n    if (s != null) {\n      minRecWrittenToEnableSkip_ = Long.parseLong(s);\n      LOG.info(\"JobConf set minRecWrittenToEnableSkip_ =\"\n          + minRecWrittenToEnableSkip_);\n    }\n  }\n\n  void addJobConfToEnvironment(JobConf jobconf, Properties env) {\n    JobConf conf = new JobConf(jobconf);\n    conf.setDeprecatedProperties();","sourceCodeStart":204,"sourceCodeEnd":240,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/PipeMapRed.java#L204-L240","documentation":"PipeMapRed's child-configuration/launch block wraps everything from FileUtil.chmod on the mapper/reducer executable, PATH resolution via PathFinder, ProcessBuilder.start(), through opening the process's stdin/stdout/stderr streams. Any IOException there is logged as 'configuration exception' and rethrown as RuntimeException(\"configuration exception\", e) with the cause attached — the task dies at setup before any record flows.","triggerScenarios":"The -mapper/-reducer command cannot be launched: executable missing from the task working directory (not shipped with -files), not executable (chmod 'a+x' failed, noexec mount, FAT filesystem), shebang interpreter missing ('No such file or directory' from exec), bad PATH resolution, or failure opening the process streams.","commonSituations":"Forgetting to ship the script (-files myscript.py) so only the name is on the command line, scripts losing the execute bit when copied through non-posix storage, task attempt dirs on noexec-mounted volumes, shebangs like '#!/usr/bin/python' where python isn't installed on nodes, Windows node path mismatches.","solutions":["Ship the executable with the job (-files /path/to/script.py) and reference it by name; verify the file appears in the task's working directory","Ensure the script has the executable bit (chmod +x before submission) and is not on a noexec mount","Fix or make the shebang portable ('#!/usr/bin/env python3'), and confirm the interpreter exists on worker nodes","Check the task log for the 'configuration exception' line and the caused-by IOException — it distinguishes chmod failure vs exec failure vs stream failure"],"exampleFix":"# before\nhadoop jar hadoop-streaming.jar -input in -output out -mapper ./my_mapper.py\n# after (ship the script so it lands in the task workdir with exec bit)\nhadoop jar hadoop-streaming.jar -input in -output out \\\n  -files hdfs://nn/tmp/my_mapper.py -mapper my_mapper.py","handlingStrategy":"try-catch","validationCode":"// preflight before submitting: the executable must exist and be runnable where the task will see it\nFile f = new File(\"my_mapper.py\");\nif (!f.exists()) throw new IllegalStateException(\"mapper script not shipped: use -files\");\nif (!f.canExecute()) throw new IllegalStateException(\"mapper script not executable: chmod +x\");\nif (!Files.getFileStore(f.toPath()).supportsFileAttributeView(PosixFileAttributeView.class)) { /* beware noexec/posix issues */ }","typeGuard":null,"tryCatchPattern":"catch RuntimeException around streaming job submission/run (JobClient.runJob) and inspect getCause(): an IOException cause from chmod/ProcessBuilder.start pinpoints launch failure; print the caused-by chain for diagnosis.","preventionTips":["Always ship executables with -files and reference them by bare name","chmod +x before submission; avoid noexec mounts for task dirs","Use portable shebangs and verify the interpreter exists on worker nodes","Smoke-test the command in a shell the way the task will run it"],"tags":["hadoop-streaming","process-launch","permissions","configuration","subprocess"],"backgroundTag":"process-launch-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}