{"record":{"id":"02f5ad96c977356a","repo":"apache/hadoop","slug":"file-lastfilestatus-getpath-lastchun","errorCode":null,"errorMessage":"File \" + lastFileStatus.getPath() + \" \" + lastChunkOffset + \",\" + lastChunkLength + \" and \" + currentFileStatus.getPath() + \" \" + currentFileStatus.getChunkOffset() + \",\" + currentFileStatus.getChunkLength() + \" are not continuous. Aborting","messagePattern":"File \" \\+ lastFileStatus\\.getPath\\(\\) \\+ \" \" \\+ lastChunkOffset \\+ \",\" \\+ lastChunkLength \\+ \" and \" \\+ currentFileStatus\\.getPath\\(\\) \\+ \" \" \\+ currentFileStatus\\.getChunkOffset\\(\\) \\+ \",\" \\+ currentFileStatus\\.getChunkLength\\(\\) \\+ \" are not continuous\\. Aborting","errorType":"exception","errorClass":"InvalidInputException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyListing.java","lineNumber":180,"sourceCode":"      long lastChunkLength = -1;\n      CopyListingFileStatus lastFileStatus = new CopyListingFileStatus();\n\n      Text currentKey = new Text();\n      Set<URI> aclSupportCheckFsSet = Sets.newHashSet();\n      Set<URI> xAttrSupportCheckFsSet = Sets.newHashSet();\n      long idx = 0;\n      while (reader.next(currentKey)) {\n        if (currentKey.equals(lastKey)) {\n          CopyListingFileStatus currentFileStatus = new CopyListingFileStatus();\n          reader.getCurrentValue(currentFileStatus);\n          if (!splitLargeFile) {\n            throw new DuplicateFileException(\"File \" + lastFileStatus.getPath()\n                + \" and \" + currentFileStatus.getPath()\n                + \" would cause duplicates. Aborting\");\n          } else {\n            if (lastChunkOffset + lastChunkLength !=\n                currentFileStatus.getChunkOffset()) {\n              throw new InvalidInputException(\"File \" + lastFileStatus.getPath()\n                  + \" \" + lastChunkOffset + \",\" + lastChunkLength\n                  + \" and \" + currentFileStatus.getPath()\n                  + \" \" + currentFileStatus.getChunkOffset() + \",\"\n                  + currentFileStatus.getChunkLength()\n                  + \" are not continuous. Aborting\");\n            }\n          }\n        }\n        reader.getCurrentValue(lastFileStatus);\n        if (context.shouldPreserve(DistCpOptions.FileAttribute.ACL)) {\n          FileSystem lastFs = lastFileStatus.getPath().getFileSystem(config);\n          URI lastFsUri = lastFs.getUri();\n          if (!aclSupportCheckFsSet.contains(lastFsUri)) {\n            DistCpUtils.checkFileSystemAclSupport(lastFs);\n            aclSupportCheckFsSet.add(lastFsUri);\n          }\n        }\n        if (context.shouldPreserve(DistCpOptions.FileAttribute.XATTR)) {","sourceCodeStart":162,"sourceCodeEnd":198,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyListing.java#L162-L198","documentation":"With -blocksperchunk (splitLargeFile), large files are emitted as chunk entries carrying (chunkOffset, chunkLength). The listing validator requires consecutive entries of the same file to be perfectly contiguous: previous offset + length must equal the next chunk's offset. A gap or overlap aborts with InvalidInputException('... are not continuous. Aborting'), because the chunked copy reassembles the file on the target via concat in exactly that order.","triggerScenarios":"Reusing a -f file listing generated under a different -blocksperchunk value or against a different version of the source file; the source file appended or rewritten between listing generation and validation; a hand-edited or partially written listing; listing and copy run with mismatched options.","commonSituations":"Retry scripts that regenerate data but reuse a stale listing; concurrent writers on the source tree during distcp planning; listings produced by an older distcp version or different chunk size; interrupted listing jobs leaving truncated sequence files.","solutions":["Discard the -f listing and let distcp regenerate it in the same run that uses -blocksperchunk","Keep the source quiescent (no appends/rewrites) between listing generation and copy","If a prebuilt listing is mandatory, rebuild it with the same -blocksperchunk value immediately before the job","Verify the listing per file: chunk offsets must read 0, L1, L1+L2, ... in entry order"],"exampleFix":"# before: old-listing was built earlier / with a different chunk size\nhadoop distcp -blocksperchunk 256 -f old-listing hdfs://nn/dst\n# -> File hdfs://src/big.dat 0,268435456 and ... are not continuous. Aborting\n\n# after: single run, fresh listing\nhadoop distcp -blocksperchunk 256 hdfs://src hdfs://nn/dst","handlingStrategy":"try-catch","validationCode":"// optional: verify chunk contiguity of a prebuilt listing sequence file\nText key = new Text();\nCopyListingFileStatus val = new CopyListingFileStatus();\nText lastKey = null; long expectedOffset = 0;\nwhile (reader.next(key)) {\n  reader.getCurrentValue(val);\n  boolean sameFile = key.equals(lastKey);\n  if (sameFile && val.getChunkOffset() != expectedOffset) {\n    throw new IOException(\"listing not continuous at \" + key\n        + \": expected offset \" + expectedOffset\n        + \", found \" + val.getChunkOffset());\n  }\n  expectedOffset = sameFile\n      ? expectedOffset + val.getChunkLength()\n      : val.getChunkLength();\n  lastKey.set(key);\n}","typeGuard":null,"tryCatchPattern":"Catch RuntimeException around DistCp.execute(); on 'are not continuous', discard the -f listing entirely and regenerate it in the same run with the same -blocksperchunk — retrying the identical listing will fail identically.","preventionTips":["Treat -f listings as single-use: regenerate for every chunked run","Keep the same -blocksperchunk value between listing creation and execution","Quiesce source data (no appends/rewrites) during distcp planning"],"tags":["distcp","blocksperchunk","chunking","data-integrity","file-listing"],"backgroundTag":"chunk-sequence-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}