apache/hadoop · error · IOException

Couldn't find image file at txid ${sig.mostRecentCheckpointT

Error message

Couldn't find image file at txid ${sig.mostRecentCheckpointTxId} even though it should have just been downloaded

What it means

After downloading a checkpoint from the active NameNode, SecondaryNameNode.doMerge() looks up fsimage_<sig.mostRecentCheckpointTxId> in its own storage via findImageFile(); a null result breaks the 'just downloaded' invariant and throws IOException. It means the image that was fetched is not present in any IMAGE-capable checkpoint directory at merge time.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java:1094

          }
        }
      }
    }

  }
    
  void doMerge(
      CheckpointSignature sig, RemoteEditLogManifest manifest,
      boolean loadImage, FSImage dstImage, FSNamesystem dstNamesystem)
      throws IOException {   
    NNStorage dstStorage = dstImage.getStorage();
    
    dstStorage.setStorageInfo(sig);
    if (loadImage) {
      File file = dstStorage.findImageFile(NameNodeFile.IMAGE,
          sig.mostRecentCheckpointTxId);
      if (file == null) {
        throw new IOException("Couldn't find image file at txid " + 
            sig.mostRecentCheckpointTxId + " even though it should have " +
            "just been downloaded");
      }
      dstNamesystem.writeLock(RwLockMode.GLOBAL);
      try {
        dstImage.reloadFromImageFile(file, dstNamesystem);
      } finally {
        dstNamesystem.writeUnlock(RwLockMode.GLOBAL, "reloadFromImageFile");
      }
      dstNamesystem.imageLoadComplete();
    }
    // error simulation code for junit test
    CheckpointFaultInjector.getInstance().duringMerge();   

    Checkpointer.rollForwardByApplyingLogs(manifest, dstImage, dstNamesystem);
    // The following has the side effect of purging old fsimages/edit logs.
    dstImage.saveFSImageInAllDirs(dstNamesystem, dstImage.getLastAppliedTxId());
    if (!namenode.isRollingUpgrade()) {

View on GitHub (pinned to 2add963021)

Solutions

1. Verify the file exists: ls <checkpoint.dir>/current/fsimage_* and compare txid with the signature in the 2NN log
2. Check every checkpoint directory is writable and has space (df -h) and fix any dead directory
3. Confirm the SecondaryNameNode and NameNode run the same Hadoop release (hdfs version)
4. Restart the SecondaryNameNode to force a fresh download+merge cycle

// before doMerge, confirm the download actually landed
File img = dstStorage.findImageFile(NameNodeFile.IMAGE,
    sig.mostRecentCheckpointTxId);
if (img == null || !img.exists()) {
  throw new IOException("Download incomplete: no fsimage_"
      + sig.mostRecentCheckpointTxId + " — rerun the checkpoint");
}

// wrap the checkpoint cycle; transient storage hiccups self-heal next cycle
try {
  secondary.doCheckpoint(); // download + doMerge
} catch (IOException e) { // "Couldn't find image file at txid ..."
  LOG.error("Checkpoint merge lost its image; verifying storage and retrying", e);
  scheduleCheckpointRetry();
}

When it happens

Trigger: doMerge(sig, manifest, loadImage=true, ...) is called right after the image download in doCheckpoint; findImageFile(IMAGE, mostRecentCheckpointTxId) returns null when no checkpoint dir of IMAGE type holds that exact file name — e.g., the write of one or more targets failed, files were deleted between download and merge, or the signature's txid does not match the produced file.

Common situations: One of several checkpoint directories is full/read-only so Util.doCopies skips it while another target later fails; a concurrent cleanup or manual deletion removed fsimage_<txid>; a version/name mismatch between the NN and 2NN (e.g., mixed Hadoop versions producing different file naming).

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• checkpoint directory does not exist or is not accessible.
• Could not create directory ${curDir}
• Image transfer servlet at " + url + " failed with status cod
• cannot access checkpoint directory.
• Could not find image with txid ${txId}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/93ae9a3b2309d2a4. Report an issue: GitHub.