{"record":{"id":"ff61d84f15abe3e9","repo":"apache/hadoop","slug":"failed-to-copy-source-file-s-to-dest-file-s","errorCode":null,"errorMessage":"Failed to copy source file %s to dest file %s","messagePattern":"Failed to copy source file (.+?) to dest file (.+?)","errorType":"exception","errorClass":"UncheckedIOException","httpStatus":null,"severity":"error","filePath":"hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/ops/RenameOp.java","lineNumber":125,"sourceCode":"          .executeWith(renamePool)\n          .throwFailureWhenFinished()\n          .retry(RENAME_RETRY_TIMES)\n          .revertWith(sourceInfo -> {\n            String newDstKey = dstKey + sourceInfo.key().substring(srcKey.length());\n            storage.delete(newDstKey);\n          })\n          .run(sourceInfo -> {\n            String newDstKey = dstKey + sourceInfo.key().substring(srcKey.length());\n            LOG.debug(\"Try to rename src key {} to dest key {}\", sourceInfo.key(), newDstKey);\n\n            try {\n              if (ObjectInfo.isDir(newDstKey)) {\n                mkdir(newDstKey);\n              } else {\n                copyFile(sourceInfo.key(), newDstKey, sourceInfo.size());\n              }\n            } catch (IOException e) {\n              throw new UncheckedIOException(\n                  String.format(\"Failed to copy source file %s to dest file %s\", sourceInfo.key(),\n                      newDstKey), e);\n            }\n          });\n\n      // Delete all the source keys, since we've already copied them into destination keys.\n      storage.deleteAll(srcKey);\n    }\n  }\n\n  private void renameFile(String srcKey, String dstKey, long fileSize) {\n    if (renameObjectEnabled) {\n      storage.rename(srcKey, dstKey);\n    } else {\n      Tasks.foreach(0)\n          .throwFailureWhenFinished()\n          .retry(RENAME_RETRY_TIMES)\n          .revertWith(obj -> storage.delete(dstKey))","sourceCodeStart":107,"sourceCodeEnd":143,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/ops/RenameOp.java#L107-L143","documentation":"RenameOp#renameDir renames a TOS 'directory' by listing every child key and copy-then-deleting (object stores have no recursive rename). Each child copy runs inside Tasks.foreach with .throwFailureWhenFinished() and a revert (delete of the partial destination); if copyFile's IOException survives the retries, it is rethrown as an UncheckedIOException with this message naming the source and destination keys. The whole rename fails atomically-ish: destinations already copied are reverted, sources are never deleted (storage.deleteAll(srcKey) is only reached on success).","triggerScenarios":"fs.rename() of a directory whose child copyFile fails: server-side copyObject errors (access denied on src object, missing PutObject permission on dst, cross-storage-class or cross-bucket copy restrictions, rate limiting/throttling, transient 5xx), network interruptions during the parallel copies, or the source object disappearing mid-listing.","commonSituations":"IAM credentials with GetObject but not PutObject; TOS throttling during large directory moves run by many concurrent tasks; copying objects in archive/cold storage classes the account may not copy synchronously; flaky network between compute cluster and TOS endpoint; concurrent job deleted source files during the rename.","solutions":["Unwrap the UncheckedIOException and read the cause: TOS error codes there (403 AccessDenied, 429 throttling, 5xx) tell you whether it is permissions, quota, or transient.","Verify IAM policy grants both tos:GetObject on the source prefix and tos:PutObject/tos:DeleteObject on the destination prefix.","Retry the rename job — transient throttling/5xx copies succeed on rerun; consider lowering map/rename concurrency if you saw 429s.","Confirm source keys still exist (no concurrent deletion) and that source/destination are in copy-compatible storage classes.","Check endpoint/network health between the cluster and the TOS endpoint if all copies fail."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"if (!fs.exists(src) || !fs.exists(dst.getParent())) {\n  throw new IllegalArgumentException(\"rename source or destination parent missing\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  fs.rename(srcDir, dstDir);\n} catch (RuntimeException e) {\n  Throwable cause = (e instanceof UncheckedIOException)\n      ? ((UncheckedIOException) e).getCause() : e;\n  if (isTransient(cause)) { fs.rename(srcDir, dstDir); } // safe: sources kept on failure\n  else { throw e; }\n}","preventionTips":["Pre-flight check IAM grants: GetObject on source prefix, PutObject/DeleteObject on destination prefix.","Throttle concurrent directory renames during peak load to avoid TOS 429 responses.","Treat UncheckedIOException cause as the real TOS error code when triaging."],"tags":["rename","object-storage","tos","copy-object","unchecked-io-exception","directory-rename"],"backgroundTag":"object-copy-during-rename-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}