apache/hadoop · error · RemoteFileChangedException

File to rename disappeared during the rename operation.

Error message

File to rename disappeared during the rename operation.

What it means

During rename, the copy of each file starts with a HEAD (getObjectMetadata) guarded by once() for the source key. If that HEAD raises FileNotFoundException for an object the initial LIST had already enumerated, the object was deleted after listing and before copying; S3A wraps it as RemoteFileChangedException with reason FILE_NOT_FOUND_SINGLE_ATTEMPT ('File to rename disappeared during the rename operation.'). The S3A retry policy deliberately maps RemoteFileChangedException to fail, so it is not silently retried - the rename aborts so callers can re-evaluate the source tree.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:4581

            .newInputStreamStatistics()
            .getChangeTrackerStatistics(),
        srcAttributes);

    String action = "copyFile(" + srcKey + ", " + dstKey + ")";
    Invoker readInvoker = readContext.getReadInvoker();

    HeadObjectResponse srcom;
    try {
      srcom = once(action, srcKey,
          () ->
              getObjectMetadata(srcKey, changeTracker, readInvoker, "copy"));
    } catch (FileNotFoundException e) {
      // if rename fails at this point it means that the expected file was not
      // found.
      // This means the File was deleted since LIST enumerated it.
      LOG.debug("getObjectMetadata({}) failed to find an expected file",
          srcKey, e);
      throw new RemoteFileChangedException(
          keyToQualifiedPath(srcKey).toString(),
          action,
          RemoteFileChangedException.FILE_NOT_FOUND_SINGLE_ATTEMPT,
          e);
    }

    CopyObjectRequest.Builder copyObjectRequestBuilder =
        getRequestFactory().newCopyObjectRequestBuilder(srcKey, dstKey, srcom);
    changeTracker.maybeApplyConstraint(copyObjectRequestBuilder);
    final CopyObjectRequest copyRequest = copyObjectRequestBuilder.build();
    LOG.debug("Copy Request: {}", copyRequest);
    CopyObjectResponse response;

    // transfer manager is skipped if disabled or the file is too small to worry about
    final boolean useTransferManager = isMultipartCopyEnabled && size >= multiPartThreshold;
    if (useTransferManager) {
      // use transfer manager
      response = readInvoker.retry(

View on GitHub (pinned to 2add963021)

Solutions

1. Retry the whole rename at the application level after re-listing - the source set changed, not the destination
2. Serialize deletes and renames of the same tree (coordination, staging areas)
3. Rename only immutable, closed output trees to shrink the race window

Example fix

// before: single rename attempt on a mutating tree
boolean ok = fs.rename(src, dst);

// after: bounded retry on concurrent modification
for (int attempt = 1; attempt <= 3; attempt++) {
  try {
    if (fs.rename(src, dst)) { ok = true; break; }
  } catch (RemoteFileChangedException e) {
    LOG.warn("source tree changed, retrying rename", e);
  }
}

Catch RemoteFileChangedException around rename(): it is non-retriable at the SDK layer by design (the retry policy maps it to fail), so re-list the source and re-run the whole rename with a bounded attempt count and backoff; abort to a reconciliation step if the tree keeps changing.

When it happens

Trigger: Another client deleting objects out of the directory while rename(srcDir, dst) iterates it; job abort/cleanup racing a rename of the same tree; S3 lifecycle expiration firing during a long rename.

Common situations: Concurrent pipelines writing and deleting the same tree; job kill between listing and copy phases; very large directory renames overlapping retention policies.

Related errors

• source is root directory
• dest is root directory
• source and dest refer to the same file or directory
• Failed to rename <src> to <dst>; source is a directory and d
• Destination is a non-empty directory

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a575e4bd781a9e0b. Report an issue: GitHub.