{"record":{"id":"94a8ae0a84b63398","repo":"apache/hadoop","slug":"ipc-s-epoch-is-less-than-the-last-promised-epoc","errorCode":null,"errorMessage":"IPC's epoch {} is less than the last promised epoch {} ; journal id: {}","messagePattern":"IPC's epoch (.+?) is less than the last promised epoch (.+?) ; journal id: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java","lineNumber":487,"sourceCode":"    \n    updateHighestWrittenTxId(lastTxnId);\n    nextTxId = lastTxnId + 1;\n    lastJournalTimestamp = Time.now();\n  }\n\n  public void heartbeat(RequestInfo reqInfo) throws IOException {\n    checkRequest(reqInfo);\n  }\n  \n  /**\n   * Ensure that the given request is coming from the correct writer and in-order.\n   * @param reqInfo the request info\n   * @throws IOException if the request is invalid.\n   */\n  private synchronized void checkRequest(RequestInfo reqInfo) throws IOException {\n    // Invariant 25 from ZAB paper\n    if (reqInfo.getEpoch() < lastPromisedEpoch.get()) {\n      throw new IOException(\"IPC's epoch \" + reqInfo.getEpoch() +\n          \" is less than the last promised epoch \" +\n          lastPromisedEpoch.get() + \" ; journal id: \" + journalId);\n    } else if (reqInfo.getEpoch() > lastPromisedEpoch.get()) {\n      // A newer client has arrived. Fence any previous writers by updating\n      // the promise.\n      updateLastPromisedEpoch(reqInfo.getEpoch());\n    }\n    \n    // Ensure that the IPCs are arriving in-order as expected.\n    checkSync(reqInfo.getIpcSerialNumber() > currentEpochIpcSerial,\n        \"IPC serial %s from client %s was not higher than prior highest \" +\n        \"IPC serial %s ; journal id: %s\", reqInfo.getIpcSerialNumber(),\n        Server.getRemoteIp(), currentEpochIpcSerial, journalId);\n    currentEpochIpcSerial = reqInfo.getIpcSerialNumber();\n\n    if (reqInfo.hasCommittedTxId()) {\n      Preconditions.checkArgument(\n          reqInfo.getCommittedTxId() >= committedTxnId.get(),","sourceCodeStart":469,"sourceCodeEnd":505,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java#L469-L505","documentation":"Journal.checkRequest enforces that every write/heartbeat IPC carries an epoch at least as high as the last promised epoch. This IOException means the sender's epoch is strictly older — the JN has already promised a newer writer, so this client is fenced and its journal requests are rejected outright.","triggerScenarios":"A previously-active NameNode (or any QJournalProtocol client) keeps sending journal() or heartbeat() calls after another NN won a higher epoch via newEpoch — i.e., an old active that has not noticed it lost leadership.","commonSituations":"Split-brain after network partition: old active still streams edits while the new active took over; ZKFC missing or disabled so the old NN never transitions to standby; long GC pause on the old active delaying its fencing; stuck client session retrying with a cached old epoch.","solutions":["Stop the stale writer: identify the NN whose epoch is lower in the error text and transition it to standby (or kill it) — the exception is the fencing working as designed.","In HA deployments, make sure both NNs run ZKFC and the health/fencing configuration is correct so failover fences the old active automatically.","Check for network partitions or GC pauses that kept the old active unaware of the failover.","If you call the QJM protocol directly, always start a session with getJournalState + newEpoch and abort on fencing errors instead of retrying the old session."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Client-side: know your epoch vs the JN's promise before writing\nlong promised = jn.getJournalState(journalId).getLastPromisedEpoch();\nif (myEpoch < promised) {\n  // fenced: do NOT send write/heartbeat RPCs; recover via newEpoch\n}","typeGuard":"static boolean isFencedByEpoch(IOException ioe) {\n  return ioe.getMessage() != null\n      && ioe.getMessage().contains(\"is less than the last promised epoch\");\n}","tryCatchPattern":"try {\n  jn.journal(reqInfo, segTxId, firstTxId, numTxn, data);\n} catch (IOException ioe) {\n  if (isFencedByEpoch(ioe)) {\n    // terminal for this session: stop writing, drop to standby/recovery —\n    // do NOT retry the same epoch\n    transitionToStandbyAndRenewSession();\n  } else {\n    throw ioe;\n  }\n}","preventionTips":["Always deploy ZKFC with HA NameNodes so fencing is automatic and old actives transition to standby.","Treat any epoch-fencing IOException as 'I am no longer the writer' — never loop-retry.","Watch for GC pauses and network partitions that let an old active keep writing after failover."],"tags":["qjm","epoch","fencing","split-brain","stale-writer"],"backgroundTag":"epoch-fencing-violation","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}