apache/hadoop · error · IOException

The dst path={dst} already exists. The admin should delete i

Error message

The dst path={dst} already exists. The admin should delete it before submitting the initial distcp job.

What it means

pathCheckBeforeInitDistcp() runs when (re)submitting the initial distcp: it requires dst to be absent and then immediately enables snapshots on src. The message names the admin action - delete dst - because in practice dst is a stale tree left by a previous failed or killed balance run that never got cleaned. Distinguish from error 5042, which is the preCheck at job setup.

Source

Thrown at hadoop-tools/hadoop-federation-balance/src/main/java/org/apache/hadoop/tools/fedbalance/DistCpProcedure.java:459

  }

  private LocalJobStatus getCurrentLocalJob() throws IOException {
    if (localJob != null) {
      Job latestJob;
      try {
        latestJob = localJob.getCluster().getJob(JobID.forName(jobId));
      } catch (InterruptedException e) {
        throw new IOException(e);
      }
      return latestJob == null ? null : new LocalJobStatus(latestJob);
    } else {
      return null;
    }
  }

  private void pathCheckBeforeInitDistcp() throws IOException {
    if (dstFs.exists(dst)) { // clean up.
      throw new IOException("The dst path=" + dst + " already exists. The admin"
          + " should delete it before submitting the initial distcp job.");
    }
    Path snapshotPath = new Path(src,
        HdfsConstants.DOT_SNAPSHOT_DIR_SEPARATOR + CURRENT_SNAPSHOT_NAME);
    if (srcFs.exists(snapshotPath)) {
      throw new IOException("The src snapshot=" + snapshotPath +
          " already exists. The admin should delete the snapshot before"
          + " submitting the initial distcp.");
    }
    srcFs.allowSnapshot(src);
  }

  /**
   * Submit distcp job and return jobId.
   */
  private String submitDistCpJob(String srcParam, String dstParam,
      boolean useSnapshotDiff) throws IOException {
    List<String> command = new ArrayList<>();

View on GitHub (pinned to 2add963021)

Solutions

1. Delete the leftover destination tree ('hdfs dfs -rm -r <dst>'), then submit the initial distcp job again.
2. If the leftover dst holds data you need, rename it aside instead of deleting.
3. When cancelling a balance job, use the tool's cancel path so journal state and dst are cleaned consistently instead of kill -9.

// Before (re)submitting the initial distcp job
Path dst = context.getDst();
if (dst.getFileSystem(conf).exists(dst)) {
  LOG.warn("Leftover dst {} from a prior run; delete it before resubmitting", dst);
  // stop and require operator cleanup, or delete if policy allows
}

When it happens

Trigger: Re-running a fedbalance job after a prior attempt already created dst (the scheduler journal recovers into INIT_DISTCP and finds dstFs.exists(dst) true); scheduler recovery replaying a job whose output survived.

Common situations: A first balance attempt failed mid-copy and the operator re-submits without cleanup; killing the fedbalance client/process and starting over; journal recovery after a scheduler restart.

Related errors

• {dst} already exists.
• The src snapshot={snapshotPath} already exists. The admin sh
• {src} should be a directory.
• {src} shouldn't enable snapshot.
• DistCp failed. jobId={jobId} failure={failureInfo}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a96775567e0aaf8c. Report an issue: GitHub.