apache/hadoop · error · IOException

Proposed epoch {} <= last promise {} ; journal id: {}

Error message

Proposed epoch {} <= last promise {} ; journal id: {}

What it means

Journal.newEpoch is the QJM fencing handshake: a writer must propose an epoch strictly greater than every epoch the JournalNode has promised. This exception rejects a proposal that is not newer than the stored lastPromisedEpoch, protecting Invariant 25 (ZAB-style fencing) so a stale writer can never take the pen from a newer one.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java:348

   * Try to create a new epoch for this journal.
   * @param nsInfo the namespace, which is verified for consistency or used to
   * format, if the Journal has not yet been written to.
   * @param epoch the epoch to start
   * @return the status information necessary to begin recovery
   * @throws IOException if the node has already made a promise to another
   * writer with a higher epoch number, if the namespace is inconsistent,
   * or if a disk error occurs.
   */
  synchronized NewEpochResponseProto newEpoch(
      NamespaceInfo nsInfo, long epoch) throws IOException {

    checkFormatted();
    storage.checkConsistentNamespace(nsInfo);

    // Check that the new epoch being proposed is in fact newer than
    // any other that we've promised. 
    if (epoch <= getLastPromisedEpoch()) {
      throw new IOException("Proposed epoch " + epoch + " <= last promise " +
          getLastPromisedEpoch() + " ; journal id: " + journalId);
    }
    
    updateLastPromisedEpoch(epoch);
    abortCurSegment();
    
    NewEpochResponseProto.Builder builder =
        NewEpochResponseProto.newBuilder();

    EditLogFile latestFile = scanStorageForLatestEdits();

    if (latestFile != null) {
      builder.setLastSegmentTxId(latestFile.getFirstTxId());
    }
    
    return builder.build();
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Ensure the stale writer is truly stopped: check for a second active NN for this nameservice and stop it; in HA use ZKFC-managed failover rather than manual start/stop.
2. Verify system clocks on all NN and JN hosts (NTP/chrony) — epochs are time-based and skewed clocks can produce non-increasing proposals.
3. Restart the fenced NN so it fetches fresh epochs via getJournalState and proposes a correctly newer one; it will recover the journal through the normal recovery flow.
4. If a VM-cloned NN is involved, discard the clone's state and start a clean standby.

// Before proposing newEpoch, learn each JN's lastPromisedEpoch via
// getJournalState and propose strictly higher:
long maxPromised = 0;
for (AsyncLogger l : loggers) {
  maxPromised = Math.max(maxPromised, l.getJournalState().getLastPromisedEpoch());
}
long proposal = Math.max(System.currentTimeMillis(), maxPromised + 1);
// only now call newEpoch with 'proposal'

static boolean isStaleEpochProposal(IOException ioe) {
  return ioe.getMessage() != null
      && ioe.getMessage().startsWith("Proposed epoch")
      && ioe.getMessage().contains("<= last promise");
}

try {
  jn.newEpoch(nsInfo, myEpoch);
} catch (IOException ioe) {
  if (isStaleEpochProposal(ioe)) {
    // we are fenced or our clock is behind: re-fetch epochs via
    // getJournalState, propose higher, or give up writership
    refetchEpochsAndRetryOnce();
  } else {
    throw ioe;
  }
}

When it happens

Trigger: A NameNode computes an epoch at or below the JN's lastPromisedEpoch and calls newEpoch: classic stale-active scenario where an old active NN (partitioned, paused, or force-fenced) tries to write after a newer NN already won the epoch; also clock skew, since epochs are time-derived.

Common situations: Old active NN comes back after failover and tries to write again (no ZKFC or manual HA transitions); NTP drift or VM pause making the restarted NN derive a lower epoch; someone restarted an NN from a snapshot/VM clone with a stale clock; multiple NNs configured active for the same nameservice.

Related errors

• IPC's epoch {} is less than the last promised epoch {} ; jou
• Already have a finalized segment {} beginning at {} ; journa
• Unable to fence {}. Fencing failed.
• Unable to fence {}
• IPC's epoch {} is not the current writer epoch {} ; journal

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/7ad90c893f8739b9. Report an issue: GitHub.