apache/hadoop · error · IOException

Inconsistent sequence file: current chunk file {srcFileStatu

Error message

Inconsistent sequence file: current chunk file {srcFileStatus} doesnt match prior entry {lastFileStatus}

What it means

CopyCommitter streams the copy-listing sequence file and merges neighboring entries that are chunks of the same file; an entry qualifies only if its path equals the prior entry's path and its chunkOffset equals prior.chunkOffset + prior.chunkLength (contiguous). This entry broke that invariant, so the chunk set cannot be concatenated safely - the listing's chunk records are non-contiguous for a file, which points at duplicate/overlapping source specifications (the same file listed twice, so its chunk sequence restarts at offset 0), or at a listing produced abnormally. Without -i this fails the commit; with -i the set is skipped with 'skipping concat this set'.

Source

Thrown at hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java:296

            }
          }
          allChunkPaths.clear();
          lastFileStatus = null;
        } else {
          if (lastFileStatus == null) {
            lastFileStatus = new CopyListingFileStatus(srcFileStatus);
          } else {
            // Two neighboring chunks have to be consecutive ones for the same
            // file, for them to be merged
            if (!srcFileStatus.getPath().equals(lastFileStatus.getPath()) ||
                srcFileStatus.getChunkOffset() !=
                (lastFileStatus.getChunkOffset() +
                lastFileStatus.getChunkLength())) {
              String emsg = "Inconsistent sequence file: current " +
                  "chunk file " + srcFileStatus + " doesnt match prior " +
                  "entry " + lastFileStatus;
              if (!ignoreFailures) {
                throw new IOException(emsg);
              } else {
                LOG.warn(emsg + ", skipping concat this set.");
              }
            } else {
              lastFileStatus.setChunkOffset(srcFileStatus.getChunkOffset());
              lastFileStatus.setChunkLength(srcFileStatus.getChunkLength());
            }
          }
        }
      }
    } finally {
      IOUtils.closeStream(sourceReader);
    }
  }

  // This method changes the target-directories' file-attributes (owner,
  // user/group permissions, etc.) based on the corresponding source directories.
  private void preserveFileAttributesForDirectories(Configuration conf)

View on GitHub (pinned to 2add963021)

Solutions

Re-run with a clean command so a fresh listing is generated - never resume from the failed attempt's listing.
De-duplicate and de-overlap the source set: remove parent/child overlaps and repeated paths from positional args and from the -f file, so every file is listed exactly once.
Drop -blocksPerChunk if you cannot guarantee clean sources - without chunk entries there is no sequence invariant to break.
If it reproduces on a de-duplicated fresh run, capture the sequence file and report or upgrade - non-contiguous chunk metadata from a single clean listing indicates a distcp bug fixed in newer releases.
When -i was used, audit the target for stray <file>.____distcpSplit____. chunks left by skipped sets and re-run those paths.

Example fix

# before: /data/subdir overlaps /data, files under subdir are listed twice
# and the second chunk sequence restarts at offset 0 -> inconsistent sequence file
hadoop distcp -blocksPerChunk 8 -f list.txt hdfs://nn/tgt

# after: one root per subtree, no overlaps, fresh listing
hadoop distcp -blocksPerChunk 8 hdfs://nn/data hdfs://nn/tgt

Defensive patterns

Strategy: retry

Validate before calling

// Pre-flight: de-overlap and de-duplicate sources so no file is listed twice
Set<Path> roots = new HashSet<>(options.getSourcePaths());
List<Path> clean = new ArrayList<>();
for (Path p : roots) {
  boolean covered = false;
  for (Path q : roots) {
    if (!q.equals(p)
        && p.toString().startsWith(q.toString() + Path.SEPARATOR)) {
      covered = true; // p is a child of another source root
    }
  }
  if (!covered) clean.add(p);
}
options.setSourcePaths(clean);

Try / catch

try {
  job.waitForCompletion(true);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Inconsistent sequence file")) {
    // the listing's chunk metadata is unusable - never resume; rebuild from scratch
    dedupeAndDeoverlapSources();
    rerunWithFreshListing(args);
  } else {
    throw e;
  }
}

Prevention

Never pass a parent directory together with one of its children as sources.
De-duplicate -f listing files before use (sort -u equivalent).
Always run distcp with a freshly generated listing; never hand-merge or reuse listings.
If the error reproduces on a clean, de-duplicated run, report it - and check for distcp fixes in newer point releases.

When it happens

Trigger: The same file appearing twice in the listing - e.g. overlapping sources (a parent directory plus one of its children, the same path duplicated in a -f file, a glob plus an explicit path) so its chunks are listed as two interleaved or back-to-back sequences where the second restarts at chunkOffset 0; listings written by mismatched or older distcp versions; rare listing-generation bugs (check known distcp JIRAs for your version).

Common situations: backup scripts passing both /data and /data/subdir as sources; -f files assembled from multiple feeds with duplicates; migrations re-running jobs with hand-merged listing remnants; versions where chunk metadata handling had fixes available in later point releases.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/bb7420b58ce5387a. Report an issue: GitHub.