{"record":{"id":"897f8b51ee38f984","repo":"apache/hadoop","slug":"hadoop-conf-dir-not-configured","errorCode":null,"errorMessage":"HADOOP_CONF_DIR not configured","messagePattern":"HADOOP_CONF_DIR not configured","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-compat-bench/src/main/java/org/apache/hadoop/fs/compat/common/HdfsCompatShellScope.java","lineNumber":114,"sourceCode":"  }\n\n  private void copyScriptsResource(File scriptDir) throws IOException {\n    Files.createDirectories(new File(scriptDir, \"cases\").toPath());\n    copyResource(\"/misc.sh\", new File(scriptDir, \"misc.sh\"));\n    String[] cases = suite.getShellCases();\n    for (String res : cases) {\n      copyResource(\"/cases/\" + res, new File(scriptDir, \"cases/\" + res));\n    }\n  }\n\n  private void setShellLogConf(File confDir) throws IOException {\n    final String hadoopHome = System.getenv(\"HADOOP_HOME\");\n    final String hadoopConfDir = System.getenv(\"HADOOP_CONF_DIR\");\n    if ((hadoopHome == null) || hadoopHome.isEmpty()) {\n      LOG.error(\"HADOOP_HOME not configured\");\n    }\n    if ((hadoopConfDir == null) || hadoopConfDir.isEmpty()) {\n      throw new IOException(\"HADOOP_CONF_DIR not configured\");\n    }\n    File srcDir = new File(hadoopConfDir).getAbsoluteFile();\n    if (!srcDir.isDirectory()) {\n      throw new IOException(\"HADOOP_CONF_DIR is not valid: \" + srcDir);\n    }\n\n    Files.createDirectories(confDir.toPath());\n    FileUtils.copyDirectory(srcDir, confDir);\n    File logConfFile = new File(confDir, \"log4j.properties\");\n    copyResource(\"/hadoop-compat-bench-log4j.properties\", logConfFile, true);\n  }\n\n  @VisibleForTesting\n  protected void copyResource(String res, File dst) throws IOException {\n    copyResource(res, dst, false);\n  }\n\n  private void copyResource(String res, File dst, boolean overwrite)","sourceCodeStart":96,"sourceCodeEnd":132,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-compat-bench/src/main/java/org/apache/hadoop/fs/compat/common/HdfsCompatShellScope.java#L96-L132","documentation":"Before running shell cases, HdfsCompatShellScope copies the cluster's HADOOP_CONF_DIR into the sandboxed conf directory (then overrides log4j). It reads HADOOP_HOME and HADOOP_CONF_DIR from the environment: a missing HADOOP_HOME is only logged as an error, but a missing or empty HADOOP_CONF_DIR throws IOException immediately.","triggerScenarios":"Launching the compat bench's shell scope from an environment where HADOOP_CONF_DIR was never exported (only HADOOP_HOME set, or neither), or where it was exported as an empty string — typical of IDE run configurations and minimal CI containers.","commonSituations":"Running tests from IntelliJ/Eclipse where the shell environment lacks Hadoop variables; CI images that install Hadoop but only set HADOOP_HOME; ssh sessions without the standard Hadoop env scripts sourced.","solutions":["export HADOOP_CONF_DIR=/etc/hadoop (or $HADOOP_HOME/etc/hadoop) in the launching shell","For IDE runs, add HADOOP_CONF_DIR (and ideally HADOOP_HOME) to the run configuration's environment variables","In CI, set the variable in the pipeline or source $HADOOP_HOME/etc/hadoop/hadoop-env.sh before the bench"],"exampleFix":"# before\nhadoop jar compat-bench.jar ... -suite myshellsuite\n# -> IOException: HADOOP_CONF_DIR not configured\n\n# after\nexport HADOOP_HOME=/opt/hadoop\nexport HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop\nhadoop jar compat-bench.jar ... -suite myshellsuite","handlingStrategy":"validation","validationCode":"String confDir = System.getenv(\"HADOOP_CONF_DIR\");\nif (confDir == null || confDir.isEmpty()) {\n  throw new IllegalStateException(\n      \"HADOOP_CONF_DIR must be exported (pointing at the cluster config dir) \"\n      + \"before shell cases can run\");\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Centralize Hadoop environment setup in one script sourced by CI jobs and IDE run configurations alike","Document HADOOP_HOME and HADOOP_CONF_DIR as required inputs of the shell bench"],"tags":["hadoop-compat-bench","environment-variables","hadoop-conf-dir","shell-tests"],"backgroundTag":"missing-env-var","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}