apache/hadoop · error · IOException

{dst} already exists.

Error message

{dst} already exists.

What it means

The second clause of DistCpProcedure.preCheck(): the destination must NOT exist, because the initial distcp creates it and later stages compute diffs against that copy. If dstFs.exists(dst) is true, the balance job aborts before any data is moved. Distinguish it from error 5046: this check runs at procedure start (preCheck), while 5046 fires during (re)submission of the initial distcp job itself.

Source

Thrown at hadoop-tools/hadoop-federation-balance/src/main/java/org/apache/hadoop/tools/fedbalance/DistCpProcedure.java:188

      return false;
    case FINISH:
      finish();
      return true;
    default:
      throw new IOException("Unexpected stage=" + stage);
    }
  }

  /**
   * Pre check of src and dst.
   */
  void preCheck() throws IOException {
    FileStatus status = srcFs.getFileStatus(src);
    if (!status.isDirectory()) {
      throw new IOException(src + " should be a directory.");
    }
    if (dstFs.exists(dst)) {
      throw new IOException(dst + " already exists.");
    }
    if (srcFs.exists(new Path(src, HdfsConstants.DOT_SNAPSHOT_DIR))) {
      throw new IOException(src + " shouldn't enable snapshot.");
    }
    updateStage(Stage.INIT_DISTCP);
  }

  /**
   * The initial distcp. Copying src to dst.
   */
  void initDistCp() throws IOException, RetryException {
    RunningJobStatus job = getCurrentJob();
    if (job != null) {
      // the distcp has been submitted.
      if (job.isComplete()) {
        jobId = null; // unset jobId because the job is done.
        if (job.isSuccessful()) {
          updateStage(Stage.DIFF_DISTCP);

View on GitHub (pinned to 2add963021)

Solutions

1. If dst is a leftover from an earlier attempt, delete it ('hdfs dfs -rm -r <dst>') and resubmit the job.
2. Otherwise choose a fresh destination path that does not exist yet.
3. Never pre-create dst; fedbalance/distcp creates it as part of the initial copy.

Example fix

# before
hdfs fedbalance -src hdfs://ns1/data -dst hdfs://ns2/data   # /data already exists

# after
hdfs dfs -rm -r hdfs://ns2/data   # if it is a leftover
hdfs fedbalance -src hdfs://ns1/data -dst hdfs://ns2/data/incoming-run1

Path dst = new Path("hdfs://ns2/balance/data");
FileSystem dstFs = dst.getFileSystem(conf);
if (dstFs.exists(dst)) {
  throw new IllegalStateException("dst already exists; delete it or pick a fresh path before fedbalance");
}

When it happens

Trigger: Submitting a fedbalance job whose -dst already exists: leftover output from a previous failed run, a directory pre-created by provisioning tooling, or running the same balance command twice.

Common situations: Re-running a balance job that failed midway without cleaning up; operators pre-creating the destination 'to save time'; -dst accidentally pointing at existing production data.

Related errors

• {src} should be a directory.
• {src} shouldn't enable snapshot.
• The dst path={dst} already exists. The admin should delete i
• DistCp failed. jobId={jobId} failure={failureInfo}
• Final DistCp failed. Failure: {failureInfo}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/9b9cf8d735bf7b14. Report an issue: GitHub.