apache/hadoop · error · IOException

Incorrect data format for ConcatDeleteOp.

Error message

Incorrect data format for ConcatDeleteOp.

What it means

Legacy-format ConcatDeleteOp.readFields(): for layout versions without EDITLOG_OP_OPTIMIZATION the record begins with a 'length' word counting its fields, and it must be at least 3 (target, sources, timestamp). A smaller length means the bytes being parsed are not a valid pre-optimization concat record, so the parse is aborted before any field is read.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java:1312

      for(int i=0; i<srcs.length; i++) {
        info[idx++] = new DeprecatedUTF8(srcs[i]);
      }
      new ArrayWritable(DeprecatedUTF8.class, info).write(out);

      FSImageSerialization.writeLong(timestamp, out);
      
      // rpc ids
      writeRpcIds(rpcClientId, rpcCallId, out);
    }

    @Override
    void readFields(DataInputStream in, int logVersion)
        throws IOException {
      if (!NameNodeLayoutVersion.supports(
          LayoutVersion.Feature.EDITLOG_OP_OPTIMIZATION, logVersion)) {
        this.length = in.readInt();
        if (length < 3) { // trg, srcs.., timestamp
          throw new IOException("Incorrect data format " +
              "for ConcatDeleteOp.");
        }
      }
      this.trg = FSImageSerialization.readString(in);
      int srcSize = 0;
      if (NameNodeLayoutVersion.supports(
          LayoutVersion.Feature.EDITLOG_OP_OPTIMIZATION, logVersion)) {
        srcSize = in.readInt();
      } else {
        srcSize = this.length - 1 - 1; // trg and timestamp
      }
      if (srcSize < 0) {
          throw new IOException("Incorrect data format. "
              + "ConcatDeleteOp cannot have a negative number of data " +
              " sources.");
      } else if (srcSize > MAX_CONCAT_SRC) {
          throw new IOException("Incorrect data format. "
              + "ConcatDeleteOp can have at most " + MAX_CONCAT_SRC +

View on GitHub (pinned to 2add963021)

Solutions

1. Run 'hdfs namenode -recover' to skip the damaged record and continue replay, then checkpoint
2. Find the corrupt txid with 'hdfs offlineEditsViewer -i <edits> -o out.xml'
3. Restore the segment from a healthy journal copy (QJM majority) or restart from the last fsimage checkpoint
4. If the segment is from an unsupported ancient release, replay it once with the release that wrote it, save a checkpoint, then upgrade

hdfs offlineEditsViewer -i /dfs/name/current/edits_0000000000000000001-0000000000000000500 -o /dev/null
# exits non-zero at the first malformed record and prints its txid

try {
  reader.readOp(); // legacy-format replay
} catch (IOException e) {
  // malformed legacy record: switch to recovery mode
  // 'hdfs namenode -recover' maps to readOp(skipBrokenEdits=true)
}

When it happens

Trigger: Replaying an old-format edits segment whose length word is corrupt, or whose bytes are misaligned because an earlier record was truncated or mis-parsed; also when a non-edits file is fed to an edits reader.

Common situations: Upgrades from very old Hadoop releases that still wrote length-prefixed fields; edits damaged by a crash mid-write or failing disk; name directory restored from a bad backup with mixed segments.

Related errors

• Incorrect data format. ConcatDeleteOp cannot have a negative
• Incorrect data format. Old rename operation.
• Incorrect data format. delete operation.
• Incorrect data format. Mkdir operation.
• Incorrect data format. times operation.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1d95cfb20277cd60. Report an issue: GitHub.