apache/hadoop · error · IOException

No logs to roll forward from lastApplied

Error message

No logs to roll forward from lastApplied

What it means

Thrown by the backup node's Checkpointer during doCheckpoint() while it tries to roll edits forward from the active NameNode. After optionally downloading the newest checkpoint image (setting lastApplied to sig.mostRecentCheckpointTxId), it checks that the first edit-log segment in the active's manifest chains onto the last applied txid; if firstRemoteLog.getStartTxId() > lastApplied + 1, the transactions between the backup node's image and the oldest retained edit log are missing and no contiguous roll-forward is possible.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java:238

    boolean needReloadImage = false;
    if (!manifest.getLogs().isEmpty()) {
      RemoteEditLog firstRemoteLog = manifest.getLogs().get(0);
      // we don't have enough logs to roll forward using only logs. Need
      // to download and load the image.
      if (firstRemoteLog.getStartTxId() > lastApplied + 1) {
        LOG.info("Unable to roll forward using only logs. Downloading " +
            "image with txid " + sig.mostRecentCheckpointTxId);
        MD5Hash downloadedHash = TransferFsImage.downloadImageToStorage(
            backupNode.nnHttpAddress, sig.mostRecentCheckpointTxId, bnStorage,
            true, false);
        bnImage.saveDigestAndRenameCheckpointImage(NameNodeFile.IMAGE,
            sig.mostRecentCheckpointTxId, downloadedHash);
        lastApplied = sig.mostRecentCheckpointTxId;
        needReloadImage = true;
      }

      if (firstRemoteLog.getStartTxId() > lastApplied + 1) {
        throw new IOException("No logs to roll forward from " + lastApplied);
      }
  
      // get edits files
      for (RemoteEditLog log : manifest.getLogs()) {
        TransferFsImage.downloadEditsToStorage(
            backupNode.nnHttpAddress, log, bnStorage);
      }

      if(needReloadImage) {
        LOG.info("Loading image with txid " + sig.mostRecentCheckpointTxId);
        backupNode.namesystem.writeLock(RwLockMode.GLOBAL);
        try {
          File file = bnStorage.findImageFile(NameNodeFile.IMAGE,
              sig.mostRecentCheckpointTxId);
          bnImage.reloadFromImageFile(file, backupNode.getNamesystem());
        } finally {
          backupNode.namesystem.writeUnlock(
              RwLockMode.GLOBAL, "doCheckpointByBackupNode");

View on GitHub (pinned to 2add963021)

Solutions

1. Take a fresh checkpoint on the active NameNode (hdfs dfsadmin -safemode enter; hdfs dfsadmin -saveNamespace; hdfs dfsadmin -safemode leave) so its newest fsimage txid reaches the retained edit range, then let the backup node download that image and roll forward.
2. Re-seed the backup node: copy the active's latest fsimage_<txid> plus its .md5 into the backup node's name/current directory (or re-format and restart the backup node against the active) so lastApplied chains with the available logs.
3. Raise edit retention on the active NN (dfs.namenode.num.checkpoints.retained, dfs.namenode.max.extra.edits.segments.retained) and restart it so purging no longer outruns the backup node.
4. If the active was re-formatted (txid reset to 0), format/re-bootstrap the backup node storage as well; the two txid epochs can never be reconciled.

Example fix

# before: backup node log loops 'No logs to roll forward from 12345'
# 1) force a fresh checkpoint on the ACTIVE NameNode
hdfs dfsadmin -safemode enter
hdfs dfsadmin -saveNamespace
hdfs dfsadmin -safemode leave
# 2) restart the backup node; it downloads fsimage_<newest> and rolls forward
# alternative: copy the active's current fsimage_<txid> and .md5 into the
# backup node's name/current/ and restart it

# backup node: newest applied txid
ls /backup/name/current | grep -oP 'fsimage_\K[0-9]+' | sort -n | tail -1
# active NN: first retained edit segment
ls /nn/name/current | grep -oP 'edits_\K[0-9]+' | sort -n | head -1
# gap (cannot roll forward) iff first_edit > backup_txid + 1

try {
  checkpointer.doCheckpoint();
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("No logs to roll forward")) {
    // re-seed the backup node from the active's image or force saveNamespace; do NOT loop unchanged
  } else { throw e; }
}

When it happens

Trigger: A BackupNode/CheckpointNode runs doCheckpoint(), selects a RemoteEditLog manifest from the active NameNode, and its own lastApplied txid (or even the active's newest fsimage txid) is older than the first retained edit segment on the active. Typical triggers: the backup node was down long enough that the active purged segments after checkpointing (dfs.namenode.num.checkpoints.retained / dfs.namenode.max.extra.edits.segments.retained), or the active NameNode was re-formatted so its txids restarted while the backup kept the old epoch.

Common situations: Backup/checkpoint node left offline for days or weeks; aggressive edit-log retention settings on the active NN; active NN storage reformatted during a test/dev cycle while the backup node directory was reused; backup node pointed at the wrong active after a reinstall.

Related errors

• Only an ACTIVE node can invoke startCheckpoint.
• We expected the next batch of events to start with transacti
• The length of the feature flag section was negative at {} by
• Interrupted waiting " + timeoutMs + "ms for a quorum of node
• Timed out waiting " + timeoutMs + "ms for a quorum of nodes

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/5835ec8404c9657f. Report an issue: GitHub.