{"record":{"id":"93ae9a3b2309d2a4","repo":"apache/hadoop","slug":"couldn-t-find-image-file-at-txid-sig-mostrecentc","errorCode":null,"errorMessage":"Couldn't find image file at txid ${sig.mostRecentCheckpointTxId} even though it should have just been downloaded","messagePattern":"Couldn't find image file at txid (.+?) even though it should have just been downloaded","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java","lineNumber":1094,"sourceCode":"          }\n        }\n      }\n    }\n\n  }\n    \n  void doMerge(\n      CheckpointSignature sig, RemoteEditLogManifest manifest,\n      boolean loadImage, FSImage dstImage, FSNamesystem dstNamesystem)\n      throws IOException {   \n    NNStorage dstStorage = dstImage.getStorage();\n    \n    dstStorage.setStorageInfo(sig);\n    if (loadImage) {\n      File file = dstStorage.findImageFile(NameNodeFile.IMAGE,\n          sig.mostRecentCheckpointTxId);\n      if (file == null) {\n        throw new IOException(\"Couldn't find image file at txid \" + \n            sig.mostRecentCheckpointTxId + \" even though it should have \" +\n            \"just been downloaded\");\n      }\n      dstNamesystem.writeLock(RwLockMode.GLOBAL);\n      try {\n        dstImage.reloadFromImageFile(file, dstNamesystem);\n      } finally {\n        dstNamesystem.writeUnlock(RwLockMode.GLOBAL, \"reloadFromImageFile\");\n      }\n      dstNamesystem.imageLoadComplete();\n    }\n    // error simulation code for junit test\n    CheckpointFaultInjector.getInstance().duringMerge();   \n\n    Checkpointer.rollForwardByApplyingLogs(manifest, dstImage, dstNamesystem);\n    // The following has the side effect of purging old fsimages/edit logs.\n    dstImage.saveFSImageInAllDirs(dstNamesystem, dstImage.getLastAppliedTxId());\n    if (!namenode.isRollingUpgrade()) {","sourceCodeStart":1076,"sourceCodeEnd":1112,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java#L1076-L1112","documentation":"After downloading a checkpoint from the active NameNode, SecondaryNameNode.doMerge() looks up fsimage_<sig.mostRecentCheckpointTxId> in its own storage via findImageFile(); a null result breaks the 'just downloaded' invariant and throws IOException. It means the image that was fetched is not present in any IMAGE-capable checkpoint directory at merge time.","triggerScenarios":"doMerge(sig, manifest, loadImage=true, ...) is called right after the image download in doCheckpoint; findImageFile(IMAGE, mostRecentCheckpointTxId) returns null when no checkpoint dir of IMAGE type holds that exact file name — e.g., the write of one or more targets failed, files were deleted between download and merge, or the signature's txid does not match the produced file.","commonSituations":"One of several checkpoint directories is full/read-only so Util.doCopies skips it while another target later fails; a concurrent cleanup or manual deletion removed fsimage_<txid>; a version/name mismatch between the NN and 2NN (e.g., mixed Hadoop versions producing different file naming).","solutions":["Verify the file exists: ls <checkpoint.dir>/current/fsimage_* and compare txid with the signature in the 2NN log","Check every checkpoint directory is writable and has space (df -h) and fix any dead directory","Confirm the SecondaryNameNode and NameNode run the same Hadoop release (hdfs version)","Restart the SecondaryNameNode to force a fresh download+merge cycle"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// before doMerge, confirm the download actually landed\nFile img = dstStorage.findImageFile(NameNodeFile.IMAGE,\n    sig.mostRecentCheckpointTxId);\nif (img == null || !img.exists()) {\n  throw new IOException(\"Download incomplete: no fsimage_\"\n      + sig.mostRecentCheckpointTxId + \" — rerun the checkpoint\");\n}","typeGuard":null,"tryCatchPattern":"// wrap the checkpoint cycle; transient storage hiccups self-heal next cycle\ntry {\n  secondary.doCheckpoint(); // download + doMerge\n} catch (IOException e) { // \"Couldn't find image file at txid ...\"\n  LOG.error(\"Checkpoint merge lost its image; verifying storage and retrying\", e);\n  scheduleCheckpointRetry();\n}","preventionTips":["Keep every checkpoint dir writable and monitored so downloads always have a healthy target","Pin NN and 2NN to the same Hadoop release so image naming never diverges","Investigate immediately if this fires twice in a row — it usually means real storage trouble, not a race"],"tags":["hdfs","secondary-namenode","checkpoint","file-not-found","merge","storage"],"backgroundTag":"file-not-found","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}