{"record":{"id":"1d95cfb20277cd60","repo":"apache/hadoop","slug":"incorrect-data-format-for-concatdeleteop","errorCode":null,"errorMessage":"Incorrect data format for ConcatDeleteOp.","messagePattern":"Incorrect data format for ConcatDeleteOp\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java","lineNumber":1312,"sourceCode":"      for(int i=0; i<srcs.length; i++) {\n        info[idx++] = new DeprecatedUTF8(srcs[i]);\n      }\n      new ArrayWritable(DeprecatedUTF8.class, info).write(out);\n\n      FSImageSerialization.writeLong(timestamp, out);\n      \n      // rpc ids\n      writeRpcIds(rpcClientId, rpcCallId, out);\n    }\n\n    @Override\n    void readFields(DataInputStream in, int logVersion)\n        throws IOException {\n      if (!NameNodeLayoutVersion.supports(\n          LayoutVersion.Feature.EDITLOG_OP_OPTIMIZATION, logVersion)) {\n        this.length = in.readInt();\n        if (length < 3) { // trg, srcs.., timestamp\n          throw new IOException(\"Incorrect data format \" +\n              \"for ConcatDeleteOp.\");\n        }\n      }\n      this.trg = FSImageSerialization.readString(in);\n      int srcSize = 0;\n      if (NameNodeLayoutVersion.supports(\n          LayoutVersion.Feature.EDITLOG_OP_OPTIMIZATION, logVersion)) {\n        srcSize = in.readInt();\n      } else {\n        srcSize = this.length - 1 - 1; // trg and timestamp\n      }\n      if (srcSize < 0) {\n          throw new IOException(\"Incorrect data format. \"\n              + \"ConcatDeleteOp cannot have a negative number of data \" +\n              \" sources.\");\n      } else if (srcSize > MAX_CONCAT_SRC) {\n          throw new IOException(\"Incorrect data format. \"\n              + \"ConcatDeleteOp can have at most \" + MAX_CONCAT_SRC +","sourceCodeStart":1294,"sourceCodeEnd":1330,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java#L1294-L1330","documentation":"Legacy-format ConcatDeleteOp.readFields(): for layout versions without EDITLOG_OP_OPTIMIZATION the record begins with a 'length' word counting its fields, and it must be at least 3 (target, sources, timestamp). A smaller length means the bytes being parsed are not a valid pre-optimization concat record, so the parse is aborted before any field is read.","triggerScenarios":"Replaying an old-format edits segment whose length word is corrupt, or whose bytes are misaligned because an earlier record was truncated or mis-parsed; also when a non-edits file is fed to an edits reader.","commonSituations":"Upgrades from very old Hadoop releases that still wrote length-prefixed fields; edits damaged by a crash mid-write or failing disk; name directory restored from a bad backup with mixed segments.","solutions":["Run 'hdfs namenode -recover' to skip the damaged record and continue replay, then checkpoint","Find the corrupt txid with 'hdfs offlineEditsViewer -i <edits> -o out.xml'","Restore the segment from a healthy journal copy (QJM majority) or restart from the last fsimage checkpoint","If the segment is from an unsupported ancient release, replay it once with the release that wrote it, save a checkpoint, then upgrade"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"hdfs offlineEditsViewer -i /dfs/name/current/edits_0000000000000000001-0000000000000000500 -o /dev/null\n# exits non-zero at the first malformed record and prints its txid","typeGuard":null,"tryCatchPattern":"try {\n  reader.readOp(); // legacy-format replay\n} catch (IOException e) {\n  // malformed legacy record: switch to recovery mode\n  // 'hdfs namenode -recover' maps to readOp(skipBrokenEdits=true)\n}","preventionTips":["Checkpoint (saveNamespace) before upgrades so legacy segments are retired early","Never mix segments from different Hadoop generations in one name dir","Keep verified backups of pre-upgrade name dirs so legacy replay has a fallback copy"],"tags":["hdfs","edit-log","corruption","legacy-format"],"backgroundTag":"edit-log-corruption","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}