apache/hadoop · error · IOException

IPC's epoch {} is less than the last promised epoch {} ; jou

Error message

IPC's epoch {} is less than the last promised epoch {} ; journal id: {}

What it means

Journal.checkRequest enforces that every write/heartbeat IPC carries an epoch at least as high as the last promised epoch. This IOException means the sender's epoch is strictly older — the JN has already promised a newer writer, so this client is fenced and its journal requests are rejected outright.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java:487

    
    updateHighestWrittenTxId(lastTxnId);
    nextTxId = lastTxnId + 1;
    lastJournalTimestamp = Time.now();
  }

  public void heartbeat(RequestInfo reqInfo) throws IOException {
    checkRequest(reqInfo);
  }
  
  /**
   * Ensure that the given request is coming from the correct writer and in-order.
   * @param reqInfo the request info
   * @throws IOException if the request is invalid.
   */
  private synchronized void checkRequest(RequestInfo reqInfo) throws IOException {
    // Invariant 25 from ZAB paper
    if (reqInfo.getEpoch() < lastPromisedEpoch.get()) {
      throw new IOException("IPC's epoch " + reqInfo.getEpoch() +
          " is less than the last promised epoch " +
          lastPromisedEpoch.get() + " ; journal id: " + journalId);
    } else if (reqInfo.getEpoch() > lastPromisedEpoch.get()) {
      // A newer client has arrived. Fence any previous writers by updating
      // the promise.
      updateLastPromisedEpoch(reqInfo.getEpoch());
    }
    
    // Ensure that the IPCs are arriving in-order as expected.
    checkSync(reqInfo.getIpcSerialNumber() > currentEpochIpcSerial,
        "IPC serial %s from client %s was not higher than prior highest " +
        "IPC serial %s ; journal id: %s", reqInfo.getIpcSerialNumber(),
        Server.getRemoteIp(), currentEpochIpcSerial, journalId);
    currentEpochIpcSerial = reqInfo.getIpcSerialNumber();

    if (reqInfo.hasCommittedTxId()) {
      Preconditions.checkArgument(
          reqInfo.getCommittedTxId() >= committedTxnId.get(),

View on GitHub (pinned to 2add963021)

Solutions

Stop the stale writer: identify the NN whose epoch is lower in the error text and transition it to standby (or kill it) — the exception is the fencing working as designed.
In HA deployments, make sure both NNs run ZKFC and the health/fencing configuration is correct so failover fences the old active automatically.
Check for network partitions or GC pauses that kept the old active unaware of the failover.
If you call the QJM protocol directly, always start a session with getJournalState + newEpoch and abort on fencing errors instead of retrying the old session.

Defensive patterns

Strategy: try-catch

Validate before calling

// Client-side: know your epoch vs the JN's promise before writing
long promised = jn.getJournalState(journalId).getLastPromisedEpoch();
if (myEpoch < promised) {
  // fenced: do NOT send write/heartbeat RPCs; recover via newEpoch
}

Type guard

static boolean isFencedByEpoch(IOException ioe) {
  return ioe.getMessage() != null
      && ioe.getMessage().contains("is less than the last promised epoch");
}

Try / catch

try {
  jn.journal(reqInfo, segTxId, firstTxId, numTxn, data);
} catch (IOException ioe) {
  if (isFencedByEpoch(ioe)) {
    // terminal for this session: stop writing, drop to standby/recovery —
    // do NOT retry the same epoch
    transitionToStandbyAndRenewSession();
  } else {
    throw ioe;
  }
}

Prevention

Always deploy ZKFC with HA NameNodes so fencing is automatic and old actives transition to standby.
Treat any epoch-fencing IOException as 'I am no longer the writer' — never loop-retry.
Watch for GC pauses and network partitions that let an old active keep writing after failover.

When it happens

Trigger: A previously-active NameNode (or any QJournalProtocol client) keeps sending journal() or heartbeat() calls after another NN won a higher epoch via newEpoch — i.e., an old active that has not noticed it lost leadership.

Common situations: Split-brain after network partition: old active still streams edits while the new active took over; ZKFC missing or disabled so the old NN never transitions to standby; long GC pause on the old active delaying its fencing; stuck client session retrying with a cached old epoch.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/94a8ae0a84b63398. Report an issue: GitHub.