apache/hadoop · error · IOException

Exception while processing StorageDirectory " + sd.getRoot()

Error message

Exception while processing StorageDirectory " + sd.getRoot()

What it means

NNStorage.getDirectories(dirType) walks each StorageDirectory of the requested type and converts its root with Util.fileAsURI; if that conversion fails - most commonly a relative local path, which fileAsURI refuses - the failure is wrapped with the offending directory root in the message. It typically surfaces while collecting IMAGE or EDITS directory lists for checkpoint or transfer setup.

Solutions

1. Make every name/edits directory absolute: /data/nn or file:///data/nn.
2. Check ${...} expansion in the XML for the exact directory named in the message.
3. After fixing, verify with 'hdfs getconf -nameDirs' and 'hdfs getconf -backupNodes'... or simply re-run getconf -confKey to confirm the effective value.

Example fix

// before - relative path
<property>
  <name>dfs.namenode.name.dir</name>
  <value>dfs/name</value>
</property>

// after - absolute path
<property>
  <name>dfs.namenode.name.dir</name>
  <value>file:///data/dfs/name</value>
</property>

// Guard every configured storage dir is resolvable to an absolute file URI
for (String loc : conf.getTrimmedStrings("dfs.namenode.name.dir")) {
  File f = new File(loc);
  boolean absolute = f.isAbsolute() || (loc.startsWith("file:"));
  if (!absolute) {
    throw new IllegalArgumentException("Relative storage dir rejected: " + loc
        + " - use /abs/path or file:///abs/path");
  }
}

static boolean isAbsoluteStorageDir(String location) {
  if (location == null) return false;
  if (location.startsWith("file:")) return URI.create(location).getPath().startsWith("/");
  return new File(location).isAbsolute();
}

When it happens

Trigger: dfs.namenode.name.dir or dfs.namenode.edits.dir contains a relative path such as 'dfs/name' (no leading '/', no file: scheme), so Util.fileAsURI(sd.getRoot()) throws when the NN builds its storage or the checkpoint lists directories.

Common situations: Configs copied from unit tests that use relative dirs; ${var} substitution resolving to a relative path; NN working directory changed by a new init script, turning previously-absolute-enough paths relative.

Related errors

• BlockPoolSliceStorage.recoverTransitionRead: attempt to…
• Can not create a Path from a null string
• Can not create a Path from an empty string
• cluster Id is incompatible with others.
• Incompatible node types: storageType=

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/bde34221cd011bcd. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java:433

  /**
   * Return the list of locations being used for a specific purpose.
   * i.e. Image or edit log storage.
   *
   * @param dirType Purpose of locations requested.
   * @throws IOException
   */
  Collection<URI> getDirectories(NameNodeDirType dirType)
      throws IOException {
    ArrayList<URI> list = new ArrayList<>();
    Iterator<StorageDirectory> it = (dirType == null) ? dirIterator() :
                                    dirIterator(dirType);
    for ( ; it.hasNext();) {
      StorageDirectory sd = it.next();
      try {
        list.add(Util.fileAsURI(sd.getRoot()));
      } catch (IOException e) {
        throw new IOException("Exception while processing " +
            "StorageDirectory " + sd.getRoot(), e);
      }
    }
    return list;
  }
  
  /**
   * Determine the last transaction ID noted in this storage directory.
   * This txid is stored in a special seen_txid file since it might not
   * correspond to the latest image or edit log. For example, an image-only
   * directory will have this txid incremented when edits logs roll, even
   * though the edits logs are in a different directory.
   *
   * @param sd StorageDirectory to check
   * @return If file exists and can be read, last recorded txid. If not, 0L.
   * @throws IOException On errors processing file pointed to by sd
   */
  static long readTransactionIdFile(StorageDirectory sd) throws IOException {

View on GitHub (pinned to 2add963021)