{"record":{"id":"5835ec8404c9657f","repo":"apache/hadoop","slug":"no-logs-to-roll-forward-from-lastapplied","errorCode":null,"errorMessage":"No logs to roll forward from lastApplied","messagePattern":"No logs to roll forward from lastApplied","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java","lineNumber":238,"sourceCode":"    boolean needReloadImage = false;\n    if (!manifest.getLogs().isEmpty()) {\n      RemoteEditLog firstRemoteLog = manifest.getLogs().get(0);\n      // we don't have enough logs to roll forward using only logs. Need\n      // to download and load the image.\n      if (firstRemoteLog.getStartTxId() > lastApplied + 1) {\n        LOG.info(\"Unable to roll forward using only logs. Downloading \" +\n            \"image with txid \" + sig.mostRecentCheckpointTxId);\n        MD5Hash downloadedHash = TransferFsImage.downloadImageToStorage(\n            backupNode.nnHttpAddress, sig.mostRecentCheckpointTxId, bnStorage,\n            true, false);\n        bnImage.saveDigestAndRenameCheckpointImage(NameNodeFile.IMAGE,\n            sig.mostRecentCheckpointTxId, downloadedHash);\n        lastApplied = sig.mostRecentCheckpointTxId;\n        needReloadImage = true;\n      }\n\n      if (firstRemoteLog.getStartTxId() > lastApplied + 1) {\n        throw new IOException(\"No logs to roll forward from \" + lastApplied);\n      }\n  \n      // get edits files\n      for (RemoteEditLog log : manifest.getLogs()) {\n        TransferFsImage.downloadEditsToStorage(\n            backupNode.nnHttpAddress, log, bnStorage);\n      }\n\n      if(needReloadImage) {\n        LOG.info(\"Loading image with txid \" + sig.mostRecentCheckpointTxId);\n        backupNode.namesystem.writeLock(RwLockMode.GLOBAL);\n        try {\n          File file = bnStorage.findImageFile(NameNodeFile.IMAGE,\n              sig.mostRecentCheckpointTxId);\n          bnImage.reloadFromImageFile(file, backupNode.getNamesystem());\n        } finally {\n          backupNode.namesystem.writeUnlock(\n              RwLockMode.GLOBAL, \"doCheckpointByBackupNode\");","sourceCodeStart":220,"sourceCodeEnd":256,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java#L220-L256","documentation":"Thrown by the backup node's Checkpointer during doCheckpoint() while it tries to roll edits forward from the active NameNode. After optionally downloading the newest checkpoint image (setting lastApplied to sig.mostRecentCheckpointTxId), it checks that the first edit-log segment in the active's manifest chains onto the last applied txid; if firstRemoteLog.getStartTxId() > lastApplied + 1, the transactions between the backup node's image and the oldest retained edit log are missing and no contiguous roll-forward is possible.","triggerScenarios":"A BackupNode/CheckpointNode runs doCheckpoint(), selects a RemoteEditLog manifest from the active NameNode, and its own lastApplied txid (or even the active's newest fsimage txid) is older than the first retained edit segment on the active. Typical triggers: the backup node was down long enough that the active purged segments after checkpointing (dfs.namenode.num.checkpoints.retained / dfs.namenode.max.extra.edits.segments.retained), or the active NameNode was re-formatted so its txids restarted while the backup kept the old epoch.","commonSituations":"Backup/checkpoint node left offline for days or weeks; aggressive edit-log retention settings on the active NN; active NN storage reformatted during a test/dev cycle while the backup node directory was reused; backup node pointed at the wrong active after a reinstall.","solutions":["Take a fresh checkpoint on the active NameNode (hdfs dfsadmin -safemode enter; hdfs dfsadmin -saveNamespace; hdfs dfsadmin -safemode leave) so its newest fsimage txid reaches the retained edit range, then let the backup node download that image and roll forward.","Re-seed the backup node: copy the active's latest fsimage_<txid> plus its .md5 into the backup node's name/current directory (or re-format and restart the backup node against the active) so lastApplied chains with the available logs.","Raise edit retention on the active NN (dfs.namenode.num.checkpoints.retained, dfs.namenode.max.extra.edits.segments.retained) and restart it so purging no longer outruns the backup node.","If the active was re-formatted (txid reset to 0), format/re-bootstrap the backup node storage as well; the two txid epochs can never be reconciled."],"exampleFix":"# before: backup node log loops 'No logs to roll forward from 12345'\n# 1) force a fresh checkpoint on the ACTIVE NameNode\nhdfs dfsadmin -safemode enter\nhdfs dfsadmin -saveNamespace\nhdfs dfsadmin -safemode leave\n# 2) restart the backup node; it downloads fsimage_<newest> and rolls forward\n# alternative: copy the active's current fsimage_<txid> and .md5 into the\n# backup node's name/current/ and restart it","handlingStrategy":"try-catch","validationCode":"# backup node: newest applied txid\nls /backup/name/current | grep -oP 'fsimage_\\K[0-9]+' | sort -n | tail -1\n# active NN: first retained edit segment\nls /nn/name/current | grep -oP 'edits_\\K[0-9]+' | sort -n | head -1\n# gap (cannot roll forward) iff first_edit > backup_txid + 1","typeGuard":null,"tryCatchPattern":"try {\n  checkpointer.doCheckpoint();\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"No logs to roll forward\")) {\n    // re-seed the backup node from the active's image or force saveNamespace; do NOT loop unchanged\n  } else { throw e; }\n}","preventionTips":["Keep the backup/checkpoint node running so its lastApplied never falls behind edit purging on the active.","Raise dfs.namenode.max.extra.edits.segments.retained and dfs.namenode.num.checkpoints.retained on the active when a backup node is deployed.","Never reuse backup-node storage after formatting the active NameNode; re-seed both to the same txid epoch.","Alert on the gap: first retained edits txid on the active vs backup node image txid."],"tags":["hdfs","backup-node","checkpoint","edit-log","transaction-id"],"backgroundTag":"edit-log-transaction-id-gap","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}