{"record":{"id":"08d9534f6713db45","repo":"apache/hadoop","slug":"hadoop-conf-dir-is-not-valid-srcdir","errorCode":null,"errorMessage":"HADOOP_CONF_DIR is not valid: \" + srcDir","messagePattern":"HADOOP_CONF_DIR is not valid: \" \\+ srcDir","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-compat-bench/src/main/java/org/apache/hadoop/fs/compat/common/HdfsCompatShellScope.java","lineNumber":118,"sourceCode":"    copyResource(\"/misc.sh\", new File(scriptDir, \"misc.sh\"));\n    String[] cases = suite.getShellCases();\n    for (String res : cases) {\n      copyResource(\"/cases/\" + res, new File(scriptDir, \"cases/\" + res));\n    }\n  }\n\n  private void setShellLogConf(File confDir) throws IOException {\n    final String hadoopHome = System.getenv(\"HADOOP_HOME\");\n    final String hadoopConfDir = System.getenv(\"HADOOP_CONF_DIR\");\n    if ((hadoopHome == null) || hadoopHome.isEmpty()) {\n      LOG.error(\"HADOOP_HOME not configured\");\n    }\n    if ((hadoopConfDir == null) || hadoopConfDir.isEmpty()) {\n      throw new IOException(\"HADOOP_CONF_DIR not configured\");\n    }\n    File srcDir = new File(hadoopConfDir).getAbsoluteFile();\n    if (!srcDir.isDirectory()) {\n      throw new IOException(\"HADOOP_CONF_DIR is not valid: \" + srcDir);\n    }\n\n    Files.createDirectories(confDir.toPath());\n    FileUtils.copyDirectory(srcDir, confDir);\n    File logConfFile = new File(confDir, \"log4j.properties\");\n    copyResource(\"/hadoop-compat-bench-log4j.properties\", logConfFile, true);\n  }\n\n  @VisibleForTesting\n  protected void copyResource(String res, File dst) throws IOException {\n    copyResource(res, dst, false);\n  }\n\n  private void copyResource(String res, File dst, boolean overwrite)\n      throws IOException {\n    InputStream in = null;\n    try {\n      in = this.getClass().getResourceAsStream(res);","sourceCodeStart":100,"sourceCodeEnd":136,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-compat-bench/src/main/java/org/apache/hadoop/fs/compat/common/HdfsCompatShellScope.java#L100-L136","documentation":"HdfsCompatShellScope takes HADOOP_CONF_DIR, resolves it to an absolute File, and requires File.isDirectory() before copying it into the sandbox conf dir. isDirectory() is false both for paths that do not exist and for paths that are regular files, so both cases produce this IOException, with the offending absolute path embedded in the message.","triggerScenarios":"HADOOP_CONF_DIR pointing at a file (e.g. .../core-site.xml) instead of the directory containing it; a stale path from an uninstalled Hadoop; a symlink whose target is gone; a Docker mount that mapped a single file where a directory was expected.","commonSituations":"Env setups copied from documentation that include the filename; Hadoop upgraded or moved and the old conf directory removed; container volume mounts missing the trailing directory.","solutions":["Point HADOOP_CONF_DIR at the directory containing core-site.xml etc.: export HADOOP_CONF_DIR=/etc/hadoop","Verify with: ls -ld \"$HADOOP_CONF_DIR\" (must show a directory)","Fix broken symlinks or remount the correct directory in containers"],"exampleFix":"# before\nexport HADOOP_CONF_DIR=/etc/hadoop/core-site.xml   # a file, not a dir\n# -> IOException: HADOOP_CONF_DIR is not valid: /etc/hadoop/core-site.xml\n\n# after\nexport HADOOP_CONF_DIR=/etc/hadoop","handlingStrategy":"validation","validationCode":"File confDir = new File(System.getenv(\"HADOOP_CONF_DIR\")).getAbsoluteFile();\nif (!confDir.isDirectory()) {\n  throw new IllegalStateException(\n      \"HADOOP_CONF_DIR is not a directory: \" + confDir\n      + \" (it must contain core-site.xml etc.)\");\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Validate env-derived paths at process start rather than deep inside a test run","Derive HADOOP_CONF_DIR from HADOOP_HOME in exactly one place to avoid drift"],"tags":["hadoop-compat-bench","environment-variables","hadoop-conf-dir","invalid-path"],"backgroundTag":"invalid-config-path","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}