What causes "Hadoop home directory {} does not exist" in apache/hadoop?

Print the active value and test it: echo $HADOOP_HOME (or echo %HADOOP_HOME%) then ls that path — it must exist

apache/hadoop · error · FileNotFoundException

Hadoop home directory {} does not exist

Error message

Hadoop home directory {} does not exist

What it means

Shell.checkHadoopHome() validates the Hadoop installation directory resolved from the HADOOP_HOME environment variable or the hadoop.home.dir system property. If that path does not exist on the local filesystem, it throws FileNotFoundException during Shell's static initialization, because Hadoop cannot anchor the location of its binaries and scripts (e.g. winutils.exe on Windows). This error almost always means the environment, not the code, is wrong.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java:544

    }
    while (home.endsWith("\"")) {
      home = home.substring(0, home.length() - 1);
    }

    // after stripping any quotes, check for home dir being non-empty
    if (home.isEmpty()) {
      throw new FileNotFoundException(E_HADOOP_PROPS_EMPTY);
    }

    // check that the hadoop home dir value
    // is an absolute reference to a directory
    File homedir = new File(home);
    if (!homedir.isAbsolute()) {
      throw new FileNotFoundException("Hadoop home directory " + homedir
          + " " + E_IS_RELATIVE);
    }
    if (!homedir.exists()) {
      throw new FileNotFoundException("Hadoop home directory " + homedir
          + " " + E_DOES_NOT_EXIST);
    }
    if (!homedir.isDirectory()) {
      throw new FileNotFoundException("Hadoop home directory " + homedir
          + " "+ E_NOT_DIRECTORY);
    }
    return homedir;
  }

  /**
   * The Hadoop home directory.
   */
  private static final File HADOOP_HOME_FILE;

  /**
   * Rethrowable cause for the failure to determine the hadoop
   * home directory
   */

View on GitHub (pinned to 2add963021)

Solutions

Print the active value and test it: echo $HADOOP_HOME (or echo %HADOOP_HOME%) then ls that path — it must exist
Fix HADOOP_HOME (or the hadoop.home.dir system property) to the extracted distribution root: the directory that actually contains bin/ and etc/
In Dockerfiles, set ENV HADOOP_HOME only after COPY/extract of the distribution into that exact path
If the install location is unknown, unset HADOOP_HOME so Shell falls back to resolving home from the hadoop-common jar location on the classpath

Example fix

// before
export HADOOP_HOME=/opt/haddoop-3.3.6   # typo: directory does not exist

// after
export HADOOP_HOME=/opt/hadoop-3.3.6
ls "$HADOOP_HOME/bin" "$HADOOP_HOME/etc"   # sanity check: both must resolve

Defensive patterns

Strategy: validation

Validate before calling

String home = System.getenv("HADOOP_HOME");
if (home == null) {
  home = System.getProperty("hadoop.home.dir");
}
if (home == null || !new File(home).isAbsolute() || !new File(home).isDirectory()) {
  throw new IllegalStateException("Invalid HADOOP_HOME (must be an existing absolute directory): " + home);
}
// safe to trigger Shell / FileSystem APIs now

Try / catch

try { Shell.getHadoopHome(); } catch (FileNotFoundException e) { /* re-read env, prompt operator, abort startup */ }

Prevention

Set HADOOP_HOME in one place (entrypoint script / image ENV) right after the distribution is extracted at that exact path
Add a CI smoke step: test -d "$HADOOP_HOME/bin" before running Hadoop client tests
Fail fast at application start with a clear message instead of letting Shell's static init throw deep in a call stack
Keep hadoop.home.dir system property and HADOOP_HOME env var consistent — never set only one to a different path

When it happens

Trigger: Running any code path that touches org.apache.hadoop.util.Shell (ShellCommandExecutor, Shell.WINDOWS checks, local FileSystem setup on Windows, Shell.getHadoopHome()) while HADOOP_HOME points to a directory that is absent on this machine; or System.setProperty("hadoop.home.dir", path) with a stale/typo'd path before the first Shell call.

Common situations: Typo in HADOOP_HOME in .bashrc / Windows env vars; Hadoop tarball upgraded or moved and the old path left behind; CI or Docker images that export HADOOP_HOME before extracting the distribution; Windows dev boxes pointing at a deleted winutils folder.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/827e1f73a62e5ee6. Report an issue: GitHub.