apache/hadoop · critical · IOException

Missing directory {}

Error message

Missing directory {}

What it means

NameNodeResourceChecker's constructor registers every LOCAL edits dir (file:// entries of the edits-dir config) plus every URI in dfs.namenode.resource.checked.volumes (all marked required) as CheckedVolumes; addDirToCheck does File.exists() on each path and throws IOException('Missing directory <absolutePath>') if it does not exist. This runs during NameNode/FSNamesystem startup, so a missing dir aborts NameNode startup.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java:158

    minimumRedundantVolumes = conf.getInt(
        DFSConfigKeys.DFS_NAMENODE_CHECKED_VOLUMES_MINIMUM_KEY,
        DFSConfigKeys.DFS_NAMENODE_CHECKED_VOLUMES_MINIMUM_DEFAULT);
  }

  /**
   * Add the volume of the passed-in directory to the list of volumes to check.
   * If <code>required</code> is true, and this volume is already present, but
   * is marked redundant, it will be marked required. If the volume is already
   * present but marked required then this method is a no-op.
   * 
   * @param directoryToCheck
   *          The directory whose volume will be checked for available space.
   */
  private void addDirToCheck(URI directoryToCheck, boolean required)
      throws IOException {
    File dir = new File(directoryToCheck.getPath());
    if (!dir.exists()) {
      throw new IOException("Missing directory "+dir.getAbsolutePath());
    }
    
    CheckedVolume newVolume = new CheckedVolume(dir, required);
    CheckedVolume volume = volumes.get(newVolume.getVolume());
    if (volume == null || !volume.isRequired()) {
      volumes.put(newVolume.getVolume(), newVolume);
    }
  }

  /**
   * Return true if disk space is available on at least one of the configured
   * redundant volumes, and all of the configured required volumes.
   * 
   * @return True if the configured amount of disk space is available on at
   *         least one redundant volume and all of the required volumes, false
   *         otherwise.
   */
  public boolean hasAvailableDiskSpace() {

View on GitHub (pinned to 2add963021)

Solutions

1. Create the directory with correct ownership (`mkdir -p <path>` and `chown hdfs:hadoop <path>`) and restart the NameNode
2. Fix the URI typo in dfs.namenode.edits.dir or dfs.namenode.resource.checked.volumes
3. If the volume may legitimately be absent, remove it from dfs.namenode.resource.checked.volumes or lower dfs.namenode.resource.checked.volumes.minimum
4. Ensure mount units (fstab/systemd) are active before the NameNode service starts; pre-create dirs in provisioning so NN start is idempotent

Example fix

# before - NameNode fails at startup: IOException Missing directory /data2/dfs/edits
ls /data2   # mount missing

# after
mkdir -p /data2/dfs/edits && chown hdfs:hadoop /data2/dfs/edits
hdfs --daemon start namenode

# Preflight before NameNode start: every configured local edits/checked dir must exist
for u in $(hdfs getconf -nameDirsWithEdits 2>/dev/null | tr ',' ' '); do
  p=${u#file://}
  [ -d "$p" ] || { mkdir -p "$p" && chown hdfs:hadoop "$p"; }
done
for v in $(hdfs getconf -confKey dfs.namenode.resource.checked.volumes | tr ',' ' '); do
  [ -d "$v" ] || { echo "FATAL: checked volume $v missing - mount?"; exit 1; }
done

import java.nio.file.*;
static boolean dirReady(String p) {
  Path d = Paths.get(p);
  return Files.isDirectory(d) && Files.isWritable(d);
}

catch (IOException e) {
  if (e.getMessage().startsWith("Missing directory")) {
    // create or mount the directory, verify ownership, then retry NN start
  }
}

When it happens

Trigger: A configured local edits directory or a dfs.namenode.resource.checked.volumes path absent on disk at NameNode start - mount not mounted, path typo, directory deleted, or a fresh host where the data dir was never created.

Common situations: Disk or NFS mount missing after a host reboot; typo'd checked-volume URI; someone deleted an edits dir during cleanup; container/k8s hostPath or volume mounts not created before the NN container starts.

Related errors

• Required edits directory {} not found: dfs.namenode.edits.di
• Cannot remove current dump directory: {}
• Cannot create dump directory {}
• {} doesn't support satisfyStoragePolicy
• {} doesn't support setStoragePolicy

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e1997569cc99024b. Report an issue: GitHub.