apache/hadoop · error · IOException

Incorrect data format. ConcatDeleteOp can have at most {} so

Error message

Incorrect data format. ConcatDeleteOp can have at most {} sources, but we tried to have {} sources.

What it means

ConcatDeleteOp.readFields() caps the source count at MAX_CONCAT_SRC = 1,048,576 (FSEditLogOp.java:1250): a well-formed concat record can never exceed it, so a larger srcSize signals a corrupt length field. Note the message's 'we tried to have' prints (length-3), the legacy-derived count, not necessarily the value just read in optimized layouts.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java:1329

        if (length < 3) { // trg, srcs.., timestamp
          throw new IOException("Incorrect data format " +
              "for ConcatDeleteOp.");
        }
      }
      this.trg = FSImageSerialization.readString(in);
      int srcSize = 0;
      if (NameNodeLayoutVersion.supports(
          LayoutVersion.Feature.EDITLOG_OP_OPTIMIZATION, logVersion)) {
        srcSize = in.readInt();
      } else {
        srcSize = this.length - 1 - 1; // trg and timestamp
      }
      if (srcSize < 0) {
          throw new IOException("Incorrect data format. "
              + "ConcatDeleteOp cannot have a negative number of data " +
              " sources.");
      } else if (srcSize > MAX_CONCAT_SRC) {
          throw new IOException("Incorrect data format. "
              + "ConcatDeleteOp can have at most " + MAX_CONCAT_SRC +
              " sources, but we tried to have " + (length - 3) + " sources.");
      }
      this.srcs = new String [srcSize];
      for(int i=0; i<srcSize;i++) {
        srcs[i]= FSImageSerialization.readString(in);
      }
      
      if (NameNodeLayoutVersion.supports(
          LayoutVersion.Feature.EDITLOG_OP_OPTIMIZATION, logVersion)) {
        this.timestamp = FSImageSerialization.readLong(in);
      } else {
        this.timestamp = readLong(in);
      }
      // read RPC ids if necessary
      readRpcIds(in, logVersion);
    }

View on GitHub (pinned to 2add963021)

Solutions

Run 'hdfs namenode -recover' to skip the corrupt record, then checkpoint
Verify the damage with 'hdfs offlineEditsViewer -i <edits> -o out.xml' before deciding what to discard
Restore the segment from an intact QJM majority copy or the last fsimage

Defensive patterns

Strategy: try-catch

Validate before calling

hdfs offlineEditsViewer -i <edits segment> -o /dev/null   # pinpoints the malformed record

Try / catch

try {
  reader.readOp();
} catch (IOException e) {
  if (e.getMessage().contains("sources, but we tried to have")) {
    // corrupt srcSize field: recover by skipping or restoring a clean segment copy
  } else { throw e; }
}

Prevention

Keep multiple journal copies (QJM) so one bad length field never strands the namespace
Monitor journal disk health and replace failing drives
Checkpoint regularly to limit exposure to any single segment

When it happens

Trigger: A garbage int lands where srcSize is read (optimized layout), or a legacy length word implies more than 1M sources; both indicate corruption or a misaligned stream rather than a real client request.

Common situations: Bit rot or torn writes on journal disks; restored backups mixing segments from different epochs; extremely rarely a hand-crafted edits file.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/2fc4393f68ac6999. Report an issue: GitHub.