{"record":{"id":"2fc4393f68ac6999","repo":"apache/hadoop","slug":"incorrect-data-format-concatdeleteop-can-have-at","errorCode":null,"errorMessage":"Incorrect data format. ConcatDeleteOp can have at most {} sources, but we tried to have {} sources.","messagePattern":"Incorrect data format\\. ConcatDeleteOp can have at most (.+?) sources, but we tried to have (.+?) sources\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java","lineNumber":1329,"sourceCode":"        if (length < 3) { // trg, srcs.., timestamp\n          throw new IOException(\"Incorrect data format \" +\n              \"for ConcatDeleteOp.\");\n        }\n      }\n      this.trg = FSImageSerialization.readString(in);\n      int srcSize = 0;\n      if (NameNodeLayoutVersion.supports(\n          LayoutVersion.Feature.EDITLOG_OP_OPTIMIZATION, logVersion)) {\n        srcSize = in.readInt();\n      } else {\n        srcSize = this.length - 1 - 1; // trg and timestamp\n      }\n      if (srcSize < 0) {\n          throw new IOException(\"Incorrect data format. \"\n              + \"ConcatDeleteOp cannot have a negative number of data \" +\n              \" sources.\");\n      } else if (srcSize > MAX_CONCAT_SRC) {\n          throw new IOException(\"Incorrect data format. \"\n              + \"ConcatDeleteOp can have at most \" + MAX_CONCAT_SRC +\n              \" sources, but we tried to have \" + (length - 3) + \" sources.\");\n      }\n      this.srcs = new String [srcSize];\n      for(int i=0; i<srcSize;i++) {\n        srcs[i]= FSImageSerialization.readString(in);\n      }\n      \n      if (NameNodeLayoutVersion.supports(\n          LayoutVersion.Feature.EDITLOG_OP_OPTIMIZATION, logVersion)) {\n        this.timestamp = FSImageSerialization.readLong(in);\n      } else {\n        this.timestamp = readLong(in);\n      }\n      // read RPC ids if necessary\n      readRpcIds(in, logVersion);\n    }\n","sourceCodeStart":1311,"sourceCodeEnd":1347,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java#L1311-L1347","documentation":"ConcatDeleteOp.readFields() caps the source count at MAX_CONCAT_SRC = 1,048,576 (FSEditLogOp.java:1250): a well-formed concat record can never exceed it, so a larger srcSize signals a corrupt length field. Note the message's 'we tried to have' prints (length-3), the legacy-derived count, not necessarily the value just read in optimized layouts.","triggerScenarios":"A garbage int lands where srcSize is read (optimized layout), or a legacy length word implies more than 1M sources; both indicate corruption or a misaligned stream rather than a real client request.","commonSituations":"Bit rot or torn writes on journal disks; restored backups mixing segments from different epochs; extremely rarely a hand-crafted edits file.","solutions":["Run 'hdfs namenode -recover' to skip the corrupt record, then checkpoint","Verify the damage with 'hdfs offlineEditsViewer -i <edits> -o out.xml' before deciding what to discard","Restore the segment from an intact QJM majority copy or the last fsimage"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"hdfs offlineEditsViewer -i <edits segment> -o /dev/null   # pinpoints the malformed record","typeGuard":null,"tryCatchPattern":"try {\n  reader.readOp();\n} catch (IOException e) {\n  if (e.getMessage().contains(\"sources, but we tried to have\")) {\n    // corrupt srcSize field: recover by skipping or restoring a clean segment copy\n  } else { throw e; }\n}","preventionTips":["Keep multiple journal copies (QJM) so one bad length field never strands the namespace","Monitor journal disk health and replace failing drives","Checkpoint regularly to limit exposure to any single segment"],"tags":["hdfs","edit-log","corruption","concat","limits"],"backgroundTag":"edit-log-corruption","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}