apache/hadoop · critical · RuntimeException

configuration exception

Error message

configuration exception

What it means

PipeMapRed's child-configuration/launch block wraps everything from FileUtil.chmod on the mapper/reducer executable, PATH resolution via PathFinder, ProcessBuilder.start(), through opening the process's stdin/stdout/stderr streams. Any IOException there is logged as 'configuration exception' and rethrown as RuntimeException("configuration exception", e) with the cause attached — the task dies at setup before any record flows.

Source

Thrown at hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/PipeMapRed.java:222

      envPut(childEnv, "TMPDIR", System.getProperty("java.io.tmpdir"));

      // Start the process
      ProcessBuilder builder = new ProcessBuilder(argvSplit);
      builder.environment().putAll(childEnv.toMap());
      sim = builder.start();

      clientOut_ = new DataOutputStream(new BufferedOutputStream(
                                              sim.getOutputStream(),
                                              BUFFER_SIZE));
      clientIn_ = new DataInputStream(new BufferedInputStream(
                                              sim.getInputStream(),
                                              BUFFER_SIZE));
      clientErr_ = new DataInputStream(new BufferedInputStream(sim.getErrorStream()));
      startTime_ = System.currentTimeMillis();

    } catch (IOException e) {
      LOG.error("configuration exception", e);
      throw new RuntimeException("configuration exception", e);
    } catch (InterruptedException e)  {
      LOG.error("configuration exception", e);
      throw new RuntimeException("configuration exception", e);
    }
  }
  
  void setStreamJobDetails(JobConf job) {
    String s = job.get("stream.minRecWrittenToEnableSkip_");
    if (s != null) {
      minRecWrittenToEnableSkip_ = Long.parseLong(s);
      LOG.info("JobConf set minRecWrittenToEnableSkip_ ="
          + minRecWrittenToEnableSkip_);
    }
  }

  void addJobConfToEnvironment(JobConf jobconf, Properties env) {
    JobConf conf = new JobConf(jobconf);
    conf.setDeprecatedProperties();

View on GitHub (pinned to 2add963021)

Solutions

1. Ship the executable with the job (-files /path/to/script.py) and reference it by name; verify the file appears in the task's working directory
2. Ensure the script has the executable bit (chmod +x before submission) and is not on a noexec mount
3. Fix or make the shebang portable ('#!/usr/bin/env python3'), and confirm the interpreter exists on worker nodes
4. Check the task log for the 'configuration exception' line and the caused-by IOException — it distinguishes chmod failure vs exec failure vs stream failure

Example fix

# before
hadoop jar hadoop-streaming.jar -input in -output out -mapper ./my_mapper.py
# after (ship the script so it lands in the task workdir with exec bit)
hadoop jar hadoop-streaming.jar -input in -output out \
  -files hdfs://nn/tmp/my_mapper.py -mapper my_mapper.py

// preflight before submitting: the executable must exist and be runnable where the task will see it
File f = new File("my_mapper.py");
if (!f.exists()) throw new IllegalStateException("mapper script not shipped: use -files");
if (!f.canExecute()) throw new IllegalStateException("mapper script not executable: chmod +x");
if (!Files.getFileStore(f.toPath()).supportsFileAttributeView(PosixFileAttributeView.class)) { /* beware noexec/posix issues */ }

catch RuntimeException around streaming job submission/run (JobClient.runJob) and inspect getCause(): an IOException cause from chmod/ProcessBuilder.start pinpoints launch failure; print the caused-by chain for diagnosis.

When it happens

Trigger: The -mapper/-reducer command cannot be launched: executable missing from the task working directory (not shipped with -files), not executable (chmod 'a+x' failed, noexec mount, FAT filesystem), shebang interpreter missing ('No such file or directory' from exec), bad PATH resolution, or failure opening the process streams.

Common situations: Forgetting to ship the script (-files myscript.py) so only the name is on the command line, scripts losing the execute bit when copied through non-posix storage, task attempt dirs on noexec-mounted volumes, shebangs like '#!/usr/bin/python' where python isn't installed on nodes, Windows node path mismatches.

Related errors

• Error creating {}
• Failed to create ${basePath}[source=${source}, allow-append=
• PipeMapRed.waitOutputThreads(): subprocess failed with code
• MROutput/MRErrThread failed:
• Class not found: {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/d96f037affb70678. Report an issue: GitHub.