{"record":{"id":"2e27e69207280c31","repo":"apache/hadoop","slug":"failed-to-concat-chunk-files-for-targetfile","errorCode":null,"errorMessage":"Failed to concat chunk files for {targetFile}","messagePattern":"Failed to concat chunk files for (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java","lineNumber":274,"sourceCode":" LOG.debug(\" add \" + targetFileChunkPath + \" to concat.\");\n }\n allChunkPaths.add(targetFileChunkPath);\n if (srcFileStatus.getChunkOffset() + srcFileStatus.getChunkLength()\n == srcFileStatus.getLen()) {\n // This is the last chunk of the splits, consolidate allChunkPaths\n try {\n concatFileChunks(conf, srcFileStatus.getPath(), targetFile,\n allChunkPaths, srcFileStatus);\n } catch (IOException e) {\n // If the concat failed because a chunk file doesn't exist,\n // then we assume that the CopyMapper has skipped copying this\n // file, and we ignore the exception here.\n // If a chunk file should have been created but it was not, then\n // the CopyMapper would have failed.\n if (!isFileNotFoundException(e)) {\n String emsg = \"Failed to concat chunk files for \" + targetFile;\n if (!ignoreFailures) {\n throw new IOException(emsg, e);\n } else {\n LOG.warn(emsg, e);\n }\n }\n }\n allChunkPaths.clear();\n lastFileStatus = null;\n } else {\n if (lastFileStatus == null) {\n lastFileStatus = new CopyListingFileStatus(srcFileStatus);\n } else {\n // Two neighboring chunks have to be consecutive ones for the same\n // file, for them to be merged\n if (!srcFileStatus.getPath().equals(lastFileStatus.getPath()) ||\n srcFileStatus.getChunkOffset() !=\n (lastFileStatus.getChunkOffset() +\n lastFileStatus.getChunkLength())) {\n String emsg = \"Inconsistent sequence file: current \" +","sourceCodeStart":256,"sourceCodeEnd":292,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java#L256-L292","documentation":"With -blocksPerChunk N, large files are copied as chunk files named .____distcpSplit____. and CopyCommitter.concatFileChunks merges each chunk set back into the final file at job commit. This error means concat threw an IOException that is not a FileNotFoundException. A FileNotFound is deliberately swallowed (it implies CopyMapper skipped that file), so any other IO problem - NameNode in safe mode, permission loss, FS lacking concat support (object stores), chunks deleted mid-job - is rethrown when failures are not ignored. With -i it is only logged as a warning and the job 'succeeds' with unconcatenated chunk files left behind.","triggerScenarios":"-blocksPerChunk against a target FileSystem whose concat is unsupported (s3a://, abfs:// and similar); NameNode in safe mode or unreachable during the commit phase; another process deleting/renaming the .____distcpSplit____. chunk files between the map phase and commit; permission or ownership changes on the target directory; -i masking the failure so stray chunk files accumulate silently.","commonSituations":"large-file backups tuned with -blocksPerChunk for parallelism; commit coinciding with NN restart or safe mode; jobs combining -i with -blocksPerChunk without auditing logs; object-store targets used with options designed for HDFS-to-HDFS.","solutions":["Read the chained cause: for safe mode run 'hdfs dfsadmin -safemode leave' (or wait it out), then re-run the same distcp command - already-copied chunks are skipped and concat is re-attempted.","If the target FileSystem does not support concat (object store), drop -blocksPerChunk and re-run.","Restore permissions/ownership on the target dir so the job user can rename/concat, then retry.","If -i was set, audit job logs for 'Failed to concat chunk files' warns, and re-run those files - the target still holds .____distcpSplit____. chunks until a successful concat.","Delete stray partial chunk files for the affected target before retrying: hadoop fs -rm '.____distcpSplit____.*'."],"exampleFix":"# before: chunked copy onto a FileSystem without concat support\nhadoop distcp -blocksPerChunk 8 s3a://bucket/src hdfs://nn/tgt\n\n# after: chunk only between HDFS endpoints that support concat\nhadoop distcp -blocksPerChunk 8 hdfs://nn/src hdfs://nn/tgt","handlingStrategy":"try-catch","validationCode":"// Before enabling -blocksPerChunk, confirm the target FS is HDFS-like (supports concat)\nFileSystem tfs = targetPath.getFileSystem(conf);\nif (!(tfs instanceof DistributedFileSystem)) {\n throw new UnsupportedOperationException(\n \"-blocksPerChunk needs a FileSystem supporting concat, got \" + tfs.getUri());\n}\n// and confirm the NN is healthy before the commit phase gets there\n((DistributedFileSystem) tfs).getDataNodeStats().getDatanodeReport(DatanodeReportType.LIVE);","typeGuard":null,"tryCatchPattern":"try {\n job.waitForCompletion(true); // commit phase performs the concat\n} catch (IOException e) {\n if (e.getMessage() != null && e.getMessage().startsWith(\"Failed to concat chunk files\")) {\n IOException cause = (IOException) e.getCause();\n // act on the real cause: safe mode -> wait/leave; perms -> fix; unsupported -> drop -blocksPerChunk\n handleConcatFailure(cause);\n rerunSameDistcpCommand(); // copied chunks are skipped, concat re-attempted\n } else {\n throw e;\n }\n}","preventionTips":["Do not use -blocksPerChunk with -i (ignoreFailures) without auditing logs - failures become warnings and chunk files stay unconcatenated.","Use chunking only for HDFS-to-HDFS copies; object stores lack concat.","Verify the NameNode is out of safe mode and healthy before large chunked jobs.","After any failed chunked run, clean stray '.____distcpSplit____.*' files before retrying."],"tags":["distcp","concat","blocksperchunk","job-commit","hdfs","large-files"],"backgroundTag":"hdfs-concat-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}