{"record":{"id":"bb7420b58ce5387a","repo":"apache/hadoop","slug":"inconsistent-sequence-file-current-chunk-file-sr","errorCode":null,"errorMessage":"Inconsistent sequence file: current chunk file {srcFileStatus} doesnt match prior entry {lastFileStatus}","messagePattern":"Inconsistent sequence file: current chunk file (.+?) doesnt match prior entry (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java","lineNumber":296,"sourceCode":" }\n }\n allChunkPaths.clear();\n lastFileStatus = null;\n } else {\n if (lastFileStatus == null) {\n lastFileStatus = new CopyListingFileStatus(srcFileStatus);\n } else {\n // Two neighboring chunks have to be consecutive ones for the same\n // file, for them to be merged\n if (!srcFileStatus.getPath().equals(lastFileStatus.getPath()) ||\n srcFileStatus.getChunkOffset() !=\n (lastFileStatus.getChunkOffset() +\n lastFileStatus.getChunkLength())) {\n String emsg = \"Inconsistent sequence file: current \" +\n \"chunk file \" + srcFileStatus + \" doesnt match prior \" +\n \"entry \" + lastFileStatus;\n if (!ignoreFailures) {\n throw new IOException(emsg);\n } else {\n LOG.warn(emsg + \", skipping concat this set.\");\n }\n } else {\n lastFileStatus.setChunkOffset(srcFileStatus.getChunkOffset());\n lastFileStatus.setChunkLength(srcFileStatus.getChunkLength());\n }\n }\n }\n }\n } finally {\n IOUtils.closeStream(sourceReader);\n }\n }\n\n // This method changes the target-directories' file-attributes (owner,\n // user/group permissions, etc.) based on the corresponding source directories.\n private void preserveFileAttributesForDirectories(Configuration conf)","sourceCodeStart":278,"sourceCodeEnd":314,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java#L278-L314","documentation":"CopyCommitter streams the copy-listing sequence file and merges neighboring entries that are chunks of the same file; an entry qualifies only if its path equals the prior entry's path and its chunkOffset equals prior.chunkOffset + prior.chunkLength (contiguous). This entry broke that invariant, so the chunk set cannot be concatenated safely - the listing's chunk records are non-contiguous for a file, which points at duplicate/overlapping source specifications (the same file listed twice, so its chunk sequence restarts at offset 0), or at a listing produced abnormally. Without -i this fails the commit; with -i the set is skipped with 'skipping concat this set'.","triggerScenarios":"The same file appearing twice in the listing - e.g. overlapping sources (a parent directory plus one of its children, the same path duplicated in a -f file, a glob plus an explicit path) so its chunks are listed as two interleaved or back-to-back sequences where the second restarts at chunkOffset 0; listings written by mismatched or older distcp versions; rare listing-generation bugs (check known distcp JIRAs for your version).","commonSituations":"backup scripts passing both /data and /data/subdir as sources; -f files assembled from multiple feeds with duplicates; migrations re-running jobs with hand-merged listing remnants; versions where chunk metadata handling had fixes available in later point releases.","solutions":["Re-run with a clean command so a fresh listing is generated - never resume from the failed attempt's listing.","De-duplicate and de-overlap the source set: remove parent/child overlaps and repeated paths from positional args and from the -f file, so every file is listed exactly once.","Drop -blocksPerChunk if you cannot guarantee clean sources - without chunk entries there is no sequence invariant to break.","If it reproduces on a de-duplicated fresh run, capture the sequence file and report or upgrade - non-contiguous chunk metadata from a single clean listing indicates a distcp bug fixed in newer releases.","When -i was used, audit the target for stray .____distcpSplit____. chunks left by skipped sets and re-run those paths."],"exampleFix":"# before: /data/subdir overlaps /data, files under subdir are listed twice\n# and the second chunk sequence restarts at offset 0 -> inconsistent sequence file\nhadoop distcp -blocksPerChunk 8 -f list.txt hdfs://nn/tgt\n\n# after: one root per subtree, no overlaps, fresh listing\nhadoop distcp -blocksPerChunk 8 hdfs://nn/data hdfs://nn/tgt","handlingStrategy":"retry","validationCode":"// Pre-flight: de-overlap and de-duplicate sources so no file is listed twice\nSet roots = new HashSet<>(options.getSourcePaths());\nList clean = new ArrayList<>();\nfor (Path p : roots) {\n boolean covered = false;\n for (Path q : roots) {\n if (!q.equals(p)\n && p.toString().startsWith(q.toString() + Path.SEPARATOR)) {\n covered = true; // p is a child of another source root\n }\n }\n if (!covered) clean.add(p);\n}\noptions.setSourcePaths(clean);","typeGuard":null,"tryCatchPattern":"try {\n job.waitForCompletion(true);\n} catch (IOException e) {\n if (e.getMessage() != null && e.getMessage().startsWith(\"Inconsistent sequence file\")) {\n // the listing's chunk metadata is unusable - never resume; rebuild from scratch\n dedupeAndDeoverlapSources();\n rerunWithFreshListing(args);\n } else {\n throw e;\n }\n}","preventionTips":["Never pass a parent directory together with one of its children as sources.","De-duplicate -f listing files before use (sort -u equivalent).","Always run distcp with a freshly generated listing; never hand-merge or reuse listings.","If the error reproduces on a clean, de-duplicated run, report it - and check for distcp fixes in newer point releases."],"tags":["distcp","blocksperchunk","sequence-file","chunking","concat","listing"],"backgroundTag":"inconsistent-chunk-sequence","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}