apache/hadoop · error · IOException

HADOOP_CONF_DIR is not valid: " + srcDir

Error message

HADOOP_CONF_DIR is not valid: " + srcDir

What it means

HdfsCompatShellScope takes HADOOP_CONF_DIR, resolves it to an absolute File, and requires File.isDirectory() before copying it into the sandbox conf dir. isDirectory() is false both for paths that do not exist and for paths that are regular files, so both cases produce this IOException, with the offending absolute path embedded in the message.

Source

Thrown at hadoop-tools/hadoop-compat-bench/src/main/java/org/apache/hadoop/fs/compat/common/HdfsCompatShellScope.java:118

    copyResource("/misc.sh", new File(scriptDir, "misc.sh"));
    String[] cases = suite.getShellCases();
    for (String res : cases) {
      copyResource("/cases/" + res, new File(scriptDir, "cases/" + res));
    }
  }

  private void setShellLogConf(File confDir) throws IOException {
    final String hadoopHome = System.getenv("HADOOP_HOME");
    final String hadoopConfDir = System.getenv("HADOOP_CONF_DIR");
    if ((hadoopHome == null) || hadoopHome.isEmpty()) {
      LOG.error("HADOOP_HOME not configured");
    }
    if ((hadoopConfDir == null) || hadoopConfDir.isEmpty()) {
      throw new IOException("HADOOP_CONF_DIR not configured");
    }
    File srcDir = new File(hadoopConfDir).getAbsoluteFile();
    if (!srcDir.isDirectory()) {
      throw new IOException("HADOOP_CONF_DIR is not valid: " + srcDir);
    }

    Files.createDirectories(confDir.toPath());
    FileUtils.copyDirectory(srcDir, confDir);
    File logConfFile = new File(confDir, "log4j.properties");
    copyResource("/hadoop-compat-bench-log4j.properties", logConfFile, true);
  }

  @VisibleForTesting
  protected void copyResource(String res, File dst) throws IOException {
    copyResource(res, dst, false);
  }

  private void copyResource(String res, File dst, boolean overwrite)
      throws IOException {
    InputStream in = null;
    try {
      in = this.getClass().getResourceAsStream(res);

View on GitHub (pinned to 2add963021)

Solutions

1. Point HADOOP_CONF_DIR at the directory containing core-site.xml etc.: export HADOOP_CONF_DIR=/etc/hadoop
2. Verify with: ls -ld "$HADOOP_CONF_DIR" (must show a directory)
3. Fix broken symlinks or remount the correct directory in containers

Example fix

# before
export HADOOP_CONF_DIR=/etc/hadoop/core-site.xml   # a file, not a dir
# -> IOException: HADOOP_CONF_DIR is not valid: /etc/hadoop/core-site.xml

# after
export HADOOP_CONF_DIR=/etc/hadoop

File confDir = new File(System.getenv("HADOOP_CONF_DIR")).getAbsoluteFile();
if (!confDir.isDirectory()) {
  throw new IllegalStateException(
      "HADOOP_CONF_DIR is not a directory: " + confDir
      + " (it must contain core-site.xml etc.)");
}

When it happens

Trigger: HADOOP_CONF_DIR pointing at a file (e.g. .../core-site.xml) instead of the directory containing it; a stale path from an uninstalled Hadoop; a symlink whose target is gone; a Docker mount that mapped a single file where a directory was expected.

Common situations: Env setups copied from documentation that include the filename; Hadoop upgraded or moved and the old conf directory removed; container volume mounts missing the trailing directory.

Related errors

• HADOOP_CONF_DIR not configured
• Unsupported name: has scheme but relative path-part
• Invalid path string ${pathString}
• Invalid path string
• Cannot rename to or from /.reserved

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/08d9534f6713db45. Report an issue: GitHub.