{"record":{"id":"89e52d12261afb98","repo":"apache/hadoop","slug":"file-copy-failed-sourcepath-target","errorCode":null,"errorMessage":"File copy failed: {sourcePath} --> {target}","messagePattern":"File copy failed: (.+?) --> (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java","lineNumber":281,"sourceCode":"    String attributeString = context.getConfiguration().get(\n            DistCpOptionSwitch.PRESERVE_STATUS.getConfigLabel());\n    return DistCpUtils.unpackAttributes(attributeString);\n  }\n\n  @SuppressWarnings(\"checkstyle:parameternumber\")\n  private void copyFileWithRetry(String description,\n      CopyListingFileStatus sourceFileStatus, Path target,\n      FileStatus targrtFileStatus, Context context, FileAction action,\n      EnumSet<FileAttribute> fileAttributes, FileStatus sourceStatus)\n      throws IOException, InterruptedException {\n    long bytesCopied;\n    try {\n      bytesCopied = (Long) new RetriableFileCopyCommand(skipCrc, description,\n          action, directWrite).execute(sourceFileStatus, target, context,\n              fileAttributes, sourceStatus);\n    } catch (Exception e) {\n      context.setStatus(\"Copy Failure: \" + sourceFileStatus.getPath());\n      throw new IOException(\"File copy failed: \" + sourceFileStatus.getPath() +\n          \" --> \" + target, e);\n    }\n    incrementCounter(context, Counter.BYTESEXPECTED, sourceFileStatus.getLen());\n    incrementCounter(context, Counter.BYTESCOPIED, bytesCopied);\n    incrementCounter(context, Counter.COPY, 1);\n    totalBytesCopied += bytesCopied;\n\n    if (verboseLog) {\n      context.write(null,\n          new Text(\"FILE_COPIED: source=\" + sourceFileStatus.getPath() + \",\"\n          + \" size=\" + sourceFileStatus.getLen() + \" --> \"\n          + \"target=\" + target + \", size=\" + (targrtFileStatus == null ?\n              0 : targrtFileStatus.getLen())));\n    }\n  }\n\n  private void createTargetDirsWithRetry(String description, Path target,\n      Context context, FileStatus sourceStatus, FileSystem sourceFS) throws IOException {","sourceCodeStart":263,"sourceCodeEnd":299,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java#L263-L299","documentation":"Top-level wrapper for a single-file copy failure inside the DistCp mapper: RetriableFileCopyCommand.execute() threw after exhausting its internal retries, and CopyMapper.rethrows with the source and target paths. The underlying reason (source read error, target write error, tmp-file promote failure, length/checksum verification, interruption) is in the exception's cause chain.","triggerScenarios":"Source file unreadable or deleted mid-job (CopyReadException), target not writable or disk/quota full, promoteTmpToTarget() failing on a conflicting target file, post-copy length or checksum verification failing, or the task being interrupted.","commonSituations":"Source permissions changed or files deleted while the job ran; target quota exhausted; two distcp jobs writing the same file; flaky DataNodes/NameNode outlasting the retry budget; corrupted source blocks.","solutions":["Walk the cause chain (repeated e.getCause()) and the mapper log for the failing file to find the true error","Fix the root cause: restore source readability, free quota/space, remove conflicting target files, stop concurrent writers","Verify source integrity with 'hdfs fsck <sourcePath> -files -blocks' if block corruption is suspected","Re-run with -update: completed files are skipped and only failures are retried"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// probe that the source is openable before the job\ntry (FSDataInputStream ignored = sourceFS.open(sourcePath)) {\n  // readable\n}","typeGuard":null,"tryCatchPattern":"catch (IOException e) {  // 'File copy failed: src --> target'\n  Throwable root = e;\n  while (root.getCause() != null) root = root.getCause();\n  // root is the real failure: read error, write error, promote failure, checksum...\n  LOG.error(\"copy of {} failed: {}\", sourcePath, root, e);\n}","preventionTips":["Always re-run failed distcp jobs with -update rather than from scratch","Free target quota/space before large migrations","Ensure the job user can read all source paths (avoid mid-job permission changes)","Keep exactly one writer per target path"],"tags":["distcp","hadoop","copy","io","retry"],"backgroundTag":"file-copy-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}