{"record":{"id":"7968bc5bd1ce5deb","repo":"apache/hadoop","slug":"read-extra-bytes-after-the-terminator","errorCode":null,"errorMessage":"Read extra bytes after the terminator!","messagePattern":"Read extra bytes after the terminator!","errorType":"exception","errorClass":"java.io.IOException","httpStatus":null,"severity":"critical","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java","lineNumber":5114,"sourceCode":"      /** The end of the edit log should contain only 0x00 or 0xff bytes.\n       * If it contains other bytes, the log itself may be corrupt.\n       * It is important to check this; if we don't, a stray OP_INVALID byte \n       * could make us stop reading the edit log halfway through, and we'd never\n       * know that we had lost data.\n       */\n      limiter.clearLimit();\n      int numRead = -1, idx = 0;\n      while (true) {\n        try {\n          numRead = -1;\n          idx = 0;\n          numRead = in.read(temp);\n          if (numRead == -1) {\n            return;\n          }\n          while (idx < numRead) {\n            if ((temp[idx] != (byte)0) && (temp[idx] != (byte)-1)) {\n              throw new IOException(\"Read extra bytes after \" +\n                \"the terminator!\");\n            }\n            idx++;\n          }\n        } finally {\n          // After reading each group of bytes, we reposition the mark one\n          // byte before the next group.  Similarly, if there is an error, we\n          // want to reposition the mark one byte before the error\n          if (numRead != -1) { \n            in.reset();\n            IOUtils.skipFully(in, idx);\n            in.mark(temp.length + 1);\n            IOUtils.skipFully(in, 1);\n          }\n        }\n      }\n    }\n","sourceCodeStart":5096,"sourceCodeEnd":5132,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java#L5096-L5132","documentation":"A finalized edit segment ends with an OP_INVALID opcode followed only by 0x00/0xFF padding. After decodeOp sees OP_INVALID, verifyTerminator() scans every remaining byte and throws if anything other than 0x00 or 0xFF appears. The check exists because a stray OP_INVALID byte mid-log would otherwise silently truncate replay and lose committed transactions.","triggerScenarios":"Corruption flips a mid-record byte to 0xFF where an opcode was expected, so the reader treats it as end-of-log and then finds live record bytes during the terminator scan; or a segment was rewritten/refinalized badly so real data follows the terminator (crash during finalize, partially overwritten file).","commonSituations":"Restart after a crash during segment finalization; journal files reused/overwritten incorrectly by misconfigured shared edits dirs; underlying disk or replication damage.","solutions":["Treat this as critical corruption: run 'hdfs namenode -recover' only after confirming which transactions follow the terminator with 'hdfs offlineEditsViewer'","Restore an intact copy of the segment from the QJM majority / SecondaryNameNode instead of skipping, because skipping here risks losing committed transactions","If no clean copy exists, roll back to the previous fsimage checkpoint and re-accept the lost transactions","Fix the finalize-path root cause (single journal dir, crash loops, shared-edits misconfiguration) before reusing the journals"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"hdfs offlineEditsViewer -i <finalized edits segment> -o /dev/null   # replaying to EOF exercises the terminator path","typeGuard":null,"tryCatchPattern":"try {\n  reader.scanOp(); // or readOp; OP_INVALID triggers the terminator scan\n} catch (IOException e) {\n  if (e.getMessage().contains(\"after the terminator\")) {\n    // CRITICAL: data follows end-of-log marker -> transactions at risk\n    // prefer restoring a clean segment copy over blind recovery-mode skipping\n  } else { throw e; }\n}","preventionTips":["Never let two processes write the same edits dir; stray writes after finalize cause this","Avoid single shared edits dirs without quorum protection","After crashes during checkpoint/finalize, validate the tail of segments explicitly","Keep QJM majority copies so a clean restore beats recovery-skipping"],"tags":["hdfs","edit-log","corruption","data-loss","terminator"],"backgroundTag":"edit-log-corruption","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}