{"record":{"id":"7ad90c893f8739b9","repo":"apache/hadoop","slug":"proposed-epoch-last-promise-journal-id","errorCode":null,"errorMessage":"Proposed epoch {} <= last promise {} ; journal id: {}","messagePattern":"Proposed epoch (.+?) <= last promise (.+?) ; journal id: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java","lineNumber":348,"sourceCode":"   * Try to create a new epoch for this journal.\n   * @param nsInfo the namespace, which is verified for consistency or used to\n   * format, if the Journal has not yet been written to.\n   * @param epoch the epoch to start\n   * @return the status information necessary to begin recovery\n   * @throws IOException if the node has already made a promise to another\n   * writer with a higher epoch number, if the namespace is inconsistent,\n   * or if a disk error occurs.\n   */\n  synchronized NewEpochResponseProto newEpoch(\n      NamespaceInfo nsInfo, long epoch) throws IOException {\n\n    checkFormatted();\n    storage.checkConsistentNamespace(nsInfo);\n\n    // Check that the new epoch being proposed is in fact newer than\n    // any other that we've promised. \n    if (epoch <= getLastPromisedEpoch()) {\n      throw new IOException(\"Proposed epoch \" + epoch + \" <= last promise \" +\n          getLastPromisedEpoch() + \" ; journal id: \" + journalId);\n    }\n    \n    updateLastPromisedEpoch(epoch);\n    abortCurSegment();\n    \n    NewEpochResponseProto.Builder builder =\n        NewEpochResponseProto.newBuilder();\n\n    EditLogFile latestFile = scanStorageForLatestEdits();\n\n    if (latestFile != null) {\n      builder.setLastSegmentTxId(latestFile.getFirstTxId());\n    }\n    \n    return builder.build();\n  }\n","sourceCodeStart":330,"sourceCodeEnd":366,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java#L330-L366","documentation":"Journal.newEpoch is the QJM fencing handshake: a writer must propose an epoch strictly greater than every epoch the JournalNode has promised. This exception rejects a proposal that is not newer than the stored lastPromisedEpoch, protecting Invariant 25 (ZAB-style fencing) so a stale writer can never take the pen from a newer one.","triggerScenarios":"A NameNode computes an epoch at or below the JN's lastPromisedEpoch and calls newEpoch: classic stale-active scenario where an old active NN (partitioned, paused, or force-fenced) tries to write after a newer NN already won the epoch; also clock skew, since epochs are time-derived.","commonSituations":"Old active NN comes back after failover and tries to write again (no ZKFC or manual HA transitions); NTP drift or VM pause making the restarted NN derive a lower epoch; someone restarted an NN from a snapshot/VM clone with a stale clock; multiple NNs configured active for the same nameservice.","solutions":["Ensure the stale writer is truly stopped: check for a second active NN for this nameservice and stop it; in HA use ZKFC-managed failover rather than manual start/stop.","Verify system clocks on all NN and JN hosts (NTP/chrony) — epochs are time-based and skewed clocks can produce non-increasing proposals.","Restart the fenced NN so it fetches fresh epochs via getJournalState and proposes a correctly newer one; it will recover the journal through the normal recovery flow.","If a VM-cloned NN is involved, discard the clone's state and start a clean standby."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Before proposing newEpoch, learn each JN's lastPromisedEpoch via\n// getJournalState and propose strictly higher:\nlong maxPromised = 0;\nfor (AsyncLogger l : loggers) {\n  maxPromised = Math.max(maxPromised, l.getJournalState().getLastPromisedEpoch());\n}\nlong proposal = Math.max(System.currentTimeMillis(), maxPromised + 1);\n// only now call newEpoch with 'proposal'","typeGuard":"static boolean isStaleEpochProposal(IOException ioe) {\n  return ioe.getMessage() != null\n      && ioe.getMessage().startsWith(\"Proposed epoch\")\n      && ioe.getMessage().contains(\"<= last promise\");\n}","tryCatchPattern":"try {\n  jn.newEpoch(nsInfo, myEpoch);\n} catch (IOException ioe) {\n  if (isStaleEpochProposal(ioe)) {\n    // we are fenced or our clock is behind: re-fetch epochs via\n    // getJournalState, propose higher, or give up writership\n    refetchEpochsAndRetryOnce();\n  } else {\n    throw ioe;\n  }\n}","preventionTips":["Run NTP/chrony on all NN and JN hosts — epochs are time-derived.","In HA, let ZKFC manage every transition; never manually start a second active.","When calling the QJM protocol directly, always begin with getJournalState and propose max(clock, lastPromised+1)."],"tags":["qjm","epoch","fencing","split-brain","ha-failover"],"backgroundTag":"epoch-fencing-violation","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}