apache/hadoop · error · IOException

HADOOP_CONF_DIR not configured

Error message

HADOOP_CONF_DIR not configured

What it means

Before running shell cases, HdfsCompatShellScope copies the cluster's HADOOP_CONF_DIR into the sandboxed conf directory (then overrides log4j). It reads HADOOP_HOME and HADOOP_CONF_DIR from the environment: a missing HADOOP_HOME is only logged as an error, but a missing or empty HADOOP_CONF_DIR throws IOException immediately.

Source

Thrown at hadoop-tools/hadoop-compat-bench/src/main/java/org/apache/hadoop/fs/compat/common/HdfsCompatShellScope.java:114

  }

  private void copyScriptsResource(File scriptDir) throws IOException {
    Files.createDirectories(new File(scriptDir, "cases").toPath());
    copyResource("/misc.sh", new File(scriptDir, "misc.sh"));
    String[] cases = suite.getShellCases();
    for (String res : cases) {
      copyResource("/cases/" + res, new File(scriptDir, "cases/" + res));
    }
  }

  private void setShellLogConf(File confDir) throws IOException {
    final String hadoopHome = System.getenv("HADOOP_HOME");
    final String hadoopConfDir = System.getenv("HADOOP_CONF_DIR");
    if ((hadoopHome == null) || hadoopHome.isEmpty()) {
      LOG.error("HADOOP_HOME not configured");
    }
    if ((hadoopConfDir == null) || hadoopConfDir.isEmpty()) {
      throw new IOException("HADOOP_CONF_DIR not configured");
    }
    File srcDir = new File(hadoopConfDir).getAbsoluteFile();
    if (!srcDir.isDirectory()) {
      throw new IOException("HADOOP_CONF_DIR is not valid: " + srcDir);
    }

    Files.createDirectories(confDir.toPath());
    FileUtils.copyDirectory(srcDir, confDir);
    File logConfFile = new File(confDir, "log4j.properties");
    copyResource("/hadoop-compat-bench-log4j.properties", logConfFile, true);
  }

  @VisibleForTesting
  protected void copyResource(String res, File dst) throws IOException {
    copyResource(res, dst, false);
  }

  private void copyResource(String res, File dst, boolean overwrite)

View on GitHub (pinned to 2add963021)

Solutions

1. export HADOOP_CONF_DIR=/etc/hadoop (or $HADOOP_HOME/etc/hadoop) in the launching shell
2. For IDE runs, add HADOOP_CONF_DIR (and ideally HADOOP_HOME) to the run configuration's environment variables
3. In CI, set the variable in the pipeline or source $HADOOP_HOME/etc/hadoop/hadoop-env.sh before the bench

Example fix

# before
hadoop jar compat-bench.jar ... -suite myshellsuite
# -> IOException: HADOOP_CONF_DIR not configured

# after
export HADOOP_HOME=/opt/hadoop
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
hadoop jar compat-bench.jar ... -suite myshellsuite

String confDir = System.getenv("HADOOP_CONF_DIR");
if (confDir == null || confDir.isEmpty()) {
  throw new IllegalStateException(
      "HADOOP_CONF_DIR must be exported (pointing at the cluster config dir) "
      + "before shell cases can run");
}

When it happens

Trigger: Launching the compat bench's shell scope from an environment where HADOOP_CONF_DIR was never exported (only HADOOP_HOME set, or neither), or where it was exported as an empty string — typical of IDE run configurations and minimal CI containers.

Common situations: Running tests from IntelliJ/Eclipse where the shell environment lacks Hadoop variables; CI images that install Hadoop but only set HADOOP_HOME; ssh sessions without the standard Hadoop env scripts sourced.

Understand the failure class

Background: "environment variable is not set" and "Missing keys in environment" errors: what missing required env var messages mean and how to fix them — this error's family across 28 libraries.

Related errors

• HADOOP_CONF_DIR is not valid: " + srcDir
• Resource not found during scripts prepare: " + res
• Cannot overwrite existing resource file
• -uri is not specified.
• Cannot find any path capability

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/897f8b51ee38f984. Report an issue: GitHub.