What causes "Cannot overwrite existing resource file" in apache/hadoop?

Delete the previous sandbox/script directory between runs (rm -rf or the whole work dir)

apache/hadoop · error · IOException

Cannot overwrite existing resource file

Error message

Cannot overwrite existing resource file

What it means

copyResource refuses to clobber files it did not create: when the destination file already exists and the call was made with overwrite=false (everything except the sandbox log4j.properties, which passes overwrite=true), it throws IOException('Cannot overwrite existing resource file'). This keeps a prior run's extracted scripts from being silently mixed with a new run's.

Source

Thrown at hadoop-tools/hadoop-compat-bench/src/main/java/org/apache/hadoop/fs/compat/common/HdfsCompatShellScope.java:146

  protected void copyResource(String res, File dst) throws IOException {
    copyResource(res, dst, false);
  }

  private void copyResource(String res, File dst, boolean overwrite)
      throws IOException {
    InputStream in = null;
    try {
      in = this.getClass().getResourceAsStream(res);
      if (in == null) {
        in = this.suite.getClass().getResourceAsStream(res);
      }
      if (in == null) {
        throw new IOException("Resource not found" +
            " during scripts prepare: " + res);
      }

      if (dst.exists() && !overwrite) {
        throw new IOException("Cannot overwrite existing resource file");
      }

      Files.createDirectories(dst.getParentFile().toPath());

      byte[] buf = new byte[1024];
      try (OutputStream out = new FileOutputStream(dst)) {
        int nRead = in.read(buf);
        while (nRead != -1) {
          out.write(buf, 0, nRead);
          nRead = in.read(buf);
        }
      }
    } finally {
      if (in != null) {
        in.close();
      }
    }
  }

View on GitHub (pinned to 2add963021)

Solutions

Delete the previous sandbox/script directory between runs (rm -rf <scriptDir> or the whole work dir)
Give each run a fresh working directory instead of reusing one
Ensure only one bench run uses a given directory at a time

Example fix

# before: rerun in the same work dir
hadoop ... HdfsCompatTool -uri ... -suite myshellsuite
# -> IOException: Cannot overwrite existing resource file

# after: clean work dir first
rm -rf $COMPAT_WORKDIR && hadoop ... HdfsCompatTool -uri ... -suite myshellsuite

Defensive patterns

Strategy: validation

Validate before calling

// before preparing scripts, make sure the destination tree is clean
File scriptDir = new File(workDir, "scripts");
if (scriptDir.exists()) {
  FileUtils.deleteQuietly(scriptDir); // bench refuses to overwrite stale files
}

Prevention

Treat the bench work directory as disposable and recreate it per run
In CI, use a fresh workspace or add an explicit cleanup step before each run
Never point two concurrent bench runs at the same work directory

When it happens

Trigger: Re-running the compat shell scope into the same working/script directory where a previous run already extracted /cases/... files; a failed first attempt leaving partial files; two bench runs sharing one directory.

Common situations: Retry after a failed run without cleaning the work dir; CI workspaces that persist between builds; parallel bench runs pointed at the same scratch directory.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ef76ed120d5779c3. Report an issue: GitHub.