apache/hadoop · error · InvalidInputException

File " + lastFileStatus.getPath() + " " + lastChunkOffset +

Error message

File " + lastFileStatus.getPath() + " " + lastChunkOffset + "," + lastChunkLength + " and " + currentFileStatus.getPath() + " " + currentFileStatus.getChunkOffset() + "," + currentFileStatus.getChunkLength() + " are not continuous. Aborting

What it means

With -blocksperchunk (splitLargeFile), large files are emitted as chunk entries carrying (chunkOffset, chunkLength). The listing validator requires consecutive entries of the same file to be perfectly contiguous: previous offset + length must equal the next chunk's offset. A gap or overlap aborts with InvalidInputException('... are not continuous. Aborting'), because the chunked copy reassembles the file on the target via concat in exactly that order.

Source

Thrown at hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyListing.java:180

      long lastChunkLength = -1;
      CopyListingFileStatus lastFileStatus = new CopyListingFileStatus();

      Text currentKey = new Text();
      Set<URI> aclSupportCheckFsSet = Sets.newHashSet();
      Set<URI> xAttrSupportCheckFsSet = Sets.newHashSet();
      long idx = 0;
      while (reader.next(currentKey)) {
        if (currentKey.equals(lastKey)) {
          CopyListingFileStatus currentFileStatus = new CopyListingFileStatus();
          reader.getCurrentValue(currentFileStatus);
          if (!splitLargeFile) {
            throw new DuplicateFileException("File " + lastFileStatus.getPath()
                + " and " + currentFileStatus.getPath()
                + " would cause duplicates. Aborting");
          } else {
            if (lastChunkOffset + lastChunkLength !=
                currentFileStatus.getChunkOffset()) {
              throw new InvalidInputException("File " + lastFileStatus.getPath()
                  + " " + lastChunkOffset + "," + lastChunkLength
                  + " and " + currentFileStatus.getPath()
                  + " " + currentFileStatus.getChunkOffset() + ","
                  + currentFileStatus.getChunkLength()
                  + " are not continuous. Aborting");
            }
          }
        }
        reader.getCurrentValue(lastFileStatus);
        if (context.shouldPreserve(DistCpOptions.FileAttribute.ACL)) {
          FileSystem lastFs = lastFileStatus.getPath().getFileSystem(config);
          URI lastFsUri = lastFs.getUri();
          if (!aclSupportCheckFsSet.contains(lastFsUri)) {
            DistCpUtils.checkFileSystemAclSupport(lastFs);
            aclSupportCheckFsSet.add(lastFsUri);
          }
        }
        if (context.shouldPreserve(DistCpOptions.FileAttribute.XATTR)) {

View on GitHub (pinned to 2add963021)

Solutions

Discard the -f listing and let distcp regenerate it in the same run that uses -blocksperchunk
Keep the source quiescent (no appends/rewrites) between listing generation and copy
If a prebuilt listing is mandatory, rebuild it with the same -blocksperchunk value immediately before the job
Verify the listing per file: chunk offsets must read 0, L1, L1+L2, ... in entry order

Example fix

# before: old-listing was built earlier / with a different chunk size
hadoop distcp -blocksperchunk 256 -f old-listing hdfs://nn/dst
# -> File hdfs://src/big.dat 0,268435456 and ... are not continuous. Aborting

# after: single run, fresh listing
hadoop distcp -blocksperchunk 256 hdfs://src hdfs://nn/dst

Defensive patterns

Strategy: try-catch

Validate before calling

// optional: verify chunk contiguity of a prebuilt listing sequence file
Text key = new Text();
CopyListingFileStatus val = new CopyListingFileStatus();
Text lastKey = null; long expectedOffset = 0;
while (reader.next(key)) {
  reader.getCurrentValue(val);
  boolean sameFile = key.equals(lastKey);
  if (sameFile && val.getChunkOffset() != expectedOffset) {
    throw new IOException("listing not continuous at " + key
        + ": expected offset " + expectedOffset
        + ", found " + val.getChunkOffset());
  }
  expectedOffset = sameFile
      ? expectedOffset + val.getChunkLength()
      : val.getChunkLength();
  lastKey.set(key);
}

Try / catch

Catch RuntimeException around DistCp.execute(); on 'are not continuous', discard the -f listing entirely and regenerate it in the same run with the same -blocksperchunk — retrying the identical listing will fail identically.

Prevention

Treat -f listings as single-use: regenerate for every chunked run
Keep the same -blocksperchunk value between listing creation and execution
Quiesce source data (no appends/rewrites) during distcp planning

When it happens

Trigger: Reusing a -f file listing generated under a different -blocksperchunk value or against a different version of the source file; the source file appended or rewritten between listing generation and validation; a hand-edited or partially written listing; listing and copy run with mismatched options.

Common situations: Retry scripts that regenerate data but reuse a stale listing; concurrent writers on the source tree during distcp planning; listings produced by an older distcp version or different chunk size; interrupted listing jobs leaving truncated sequence files.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/02f5ad96c977356a. Report an issue: GitHub.