apache/hadoop · error · UncheckedIOException

Failed to copy source file %s to dest file %s

Error message

Failed to copy source file %s to dest file %s

What it means

RenameOp#renameDir renames a TOS 'directory' by listing every child key and copy-then-deleting (object stores have no recursive rename). Each child copy runs inside Tasks.foreach with .throwFailureWhenFinished() and a revert (delete of the partial destination); if copyFile's IOException survives the retries, it is rethrown as an UncheckedIOException with this message naming the source and destination keys. The whole rename fails atomically-ish: destinations already copied are reverted, sources are never deleted (storage.deleteAll(srcKey) is only reached on success).

Source

Thrown at hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/ops/RenameOp.java:125

          .executeWith(renamePool)
          .throwFailureWhenFinished()
          .retry(RENAME_RETRY_TIMES)
          .revertWith(sourceInfo -> {
            String newDstKey = dstKey + sourceInfo.key().substring(srcKey.length());
            storage.delete(newDstKey);
          })
          .run(sourceInfo -> {
            String newDstKey = dstKey + sourceInfo.key().substring(srcKey.length());
            LOG.debug("Try to rename src key {} to dest key {}", sourceInfo.key(), newDstKey);

            try {
              if (ObjectInfo.isDir(newDstKey)) {
                mkdir(newDstKey);
              } else {
                copyFile(sourceInfo.key(), newDstKey, sourceInfo.size());
              }
            } catch (IOException e) {
              throw new UncheckedIOException(
                  String.format("Failed to copy source file %s to dest file %s", sourceInfo.key(),
                      newDstKey), e);
            }
          });

      // Delete all the source keys, since we've already copied them into destination keys.
      storage.deleteAll(srcKey);
    }
  }

  private void renameFile(String srcKey, String dstKey, long fileSize) {
    if (renameObjectEnabled) {
      storage.rename(srcKey, dstKey);
    } else {
      Tasks.foreach(0)
          .throwFailureWhenFinished()
          .retry(RENAME_RETRY_TIMES)
          .revertWith(obj -> storage.delete(dstKey))

View on GitHub (pinned to 2add963021)

Solutions

1. Unwrap the UncheckedIOException and read the cause: TOS error codes there (403 AccessDenied, 429 throttling, 5xx) tell you whether it is permissions, quota, or transient.
2. Verify IAM policy grants both tos:GetObject on the source prefix and tos:PutObject/tos:DeleteObject on the destination prefix.
3. Retry the rename job — transient throttling/5xx copies succeed on rerun; consider lowering map/rename concurrency if you saw 429s.
4. Confirm source keys still exist (no concurrent deletion) and that source/destination are in copy-compatible storage classes.
5. Check endpoint/network health between the cluster and the TOS endpoint if all copies fail.

if (!fs.exists(src) || !fs.exists(dst.getParent())) {
  throw new IllegalArgumentException("rename source or destination parent missing");
}

try {
  fs.rename(srcDir, dstDir);
} catch (RuntimeException e) {
  Throwable cause = (e instanceof UncheckedIOException)
      ? ((UncheckedIOException) e).getCause() : e;
  if (isTransient(cause)) { fs.rename(srcDir, dstDir); } // safe: sources kept on failure
  else { throw e; }
}

When it happens

Trigger: fs.rename() of a directory whose child copyFile fails: server-side copyObject errors (access denied on src object, missing PutObject permission on dst, cross-storage-class or cross-bucket copy restrictions, rate limiting/throttling, transient 5xx), network interruptions during the parallel copies, or the source object disappearing mid-listing.

Common situations: IAM credentials with GetObject but not PutObject; TOS throttling during large directory moves run by many concurrent tasks; copying objects in archive/cold storage classes the account may not copy synchronously; flaky network between compute cluster and TOS endpoint; concurrent job deleted source files during the rename.

Related errors

• {} is root directory
• {} already exists
• Not supported
• Failed to rename since it is prohibited to rename dest path
• Failed to rename since the dest path %s already exists.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ff61d84f15abe3e9. Report an issue: GitHub.