What causes "Specified path ${path}does not exist" in apache/hadoop?

Create the directory before starting the daemon: mkdir -p (plus chown to the daemon user).

apache/hadoop · error · FileNotFoundException

Specified path ${path}does not exist

Error message

Specified path ${path}does not exist

What it means

DF is the wrapper around the unix 'df' command used for storage-volume accounting (capacity, usage, mount point). getMount() first checks that the wrapped java.io.File exists and throws FileNotFoundException naming the configured path when it does not - a fail-fast so daemons report a bad volume directory rather than running 'df' against nothing.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DF.java:113

  public long getAvailable() {
    return dirFile.getUsableSpace();
  }

  /** @return the amount of the volume full, as a percent. */
  public int getPercentUsed() {
    double cap = (double) getCapacity();
    double used = (cap - (double) getAvailable());
    return (int) (used * 100.0 / cap);
  }

  /**
   * @return the filesystem mount point for the indicated volume.
   * @throws IOException raised on errors performing I/O.
   */
  public String getMount() throws IOException {
    // Abort early if specified path does not exist
    if (!dirFile.exists()) {
      throw new FileNotFoundException("Specified path " + dirFile.getPath()
          + "does not exist");
    }

    if (Shell.WINDOWS) {
      // Assume a drive letter for a mount point
      this.mount = dirFile.getCanonicalPath().substring(0, 2);
    } else {
      run();
      verifyExitCode();
      parseOutput();
    }

    return mount;
  }
  
  @Override
  public String toString() {
    return

View on GitHub (pinned to 2add963021)

Solutions

Create the directory before starting the daemon: mkdir -p <dir> (plus chown to the daemon user).
Fix the configuration value (dfs.datanode.data.dir, yarn.nodemanager.local-dirs, etc.) to point at the intended existing path.
Verify with ls -ld <dir> as the daemon user; if it is a mount point, confirm the mount actually happened (df -h <dir>).

Example fix

# before
# hdfs-site.xml: dfs.datanode.data.dir = /grid/1/dn  (directory absent)
# DataNode start -> DF.getMount(): FileNotFoundException: Specified path /grid/1/dn does not exist

# after
sudo mkdir -p /grid/1/dn && sudo chown hdfs:hadoop /grid/1/dn
# then restart the DataNode

Defensive patterns

Strategy: validation

Validate before calling

java.io.File dir = new java.io.File(configuredDir);
if (!dir.exists()) {
  throw new IllegalStateException("configured dir missing: " + configuredDir
      + " - create it or fix dfs.datanode.data.dir / yarn.nodemanager.local-dirs");
}

Try / catch

try {
  String mount = df.getMount();
} catch (FileNotFoundException e) { // 'Specified path ... does not exist'
  // create the directory (mkdir -p) with correct ownership and retry
}

Prevention

Provisioning scripts must mkdir -p and chown every configured storage dir before daemon start.
Validate configured dirs in a pre-flight check at daemon startup and report all missing at once.
Re-check after mount changes - a fstab edit without creating the mountpoint will resurface this.

When it happens

Trigger: Constructing DF with a path (new DF(dir)) that does not exist on the local filesystem and calling getMount() - typically invoked by DataNode/NodeManager/local-dir management, DU usage trackers, or anything calling DF.getMount()/getDirUsage() on a configured storage directory that was never created or was deleted.

Common situations: dfs.datanode.data.dir or yarn.nodemanager.local-dirs / log dirs pointing at a directory absent on a new node; mount point not yet created in deployment scripts; directory removed by a cleanup task after the daemon started; typo in the config path.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1defe6b568b87dd4. Report an issue: GitHub.