What causes "Could not find image with txid ${txId}" in apache/hadoop?

List fsimage_ in /current and confirm the exact txid exists before retrying

apache/hadoop · error · IOException

Could not find image with txid ${txId}

Error message

Could not find image with txid ${txId}

What it means

Before pushing a checkpoint image to a peer via HTTP PUT, TransferFsImage.uploadImage() locates the local file with storage.findImageFile(nnf, txId); if no fsimage_<txId> of the requested NameNodeFile type exists locally, it throws IOException. The uploader is asked to send an image it no longer has.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java:274

      return result;
    }
    double xferSec = Math.max(
        ((float) (Time.monotonicNow() - startTime)) / 1000.0, 0.001);
    LOG.info("Uploaded image with txid " + txid + " to namenode at " + fsName
        + " in " + xferSec + " seconds");
    return TransferResult.SUCCESS;
  }

  /*
   * Uploads the imagefile using HTTP PUT method
   */
  private static void uploadImage(URL url, Configuration conf,
      NNStorage storage, NameNodeFile nnf, long txId, Canceler canceler)
      throws IOException {

    File imageFile = storage.findImageFile(nnf, txId);
    if (imageFile == null) {
      throw new IOException("Could not find image with txid " + txId);
    }

    HttpURLConnection connection = null;
    try {
      URIBuilder uriBuilder = new URIBuilder(url.toURI());

      // write all params for image upload request as query itself.
      // Request body contains the image to be uploaded.
      Map<String, String> params = ImageServlet.getParamsForPutImage(storage,
          txId, imageFile.length(), nnf);
      for (Entry<String, String> entry : params.entrySet()) {
        uriBuilder.addParameter(entry.getKey(), entry.getValue());
      }

      URL urlWithParams = uriBuilder.build().toURL();
      connection = (HttpURLConnection) connectionFactory.openConnection(
          urlWithParams, UserGroupInformation.isSecurityEnabled());
      // Set the request to PUT

View on GitHub (pinned to 2add963021)

Solutions

List fsimage_<txid> in <name.dir>/current and confirm the exact txid exists before retrying
Raise retention (dfs.image.transfer.timeout related checkpoint retention, NameNode 'dfs.namenode.num.checkpoints.retained') so files survive until upload
Ensure only one SecondaryNameNode/checkpointer uploads to this NN and no manual cleanup runs during checkpoints
Trigger a new checkpoint so a fresh image exists, then retry the transfer

Defensive patterns

Strategy: validation

Validate before calling

// before initiating the PUT, confirm the source image still exists
File imageFile = storage.findImageFile(nnf, txId);
if (imageFile == null || !imageFile.exists()) {
  LOG.warn("Image fsimage_{} vanished; skipping upload this cycle", txId);
  return; // let the next checkpoint produce a fresh image
}

Try / catch

try {
  TransferFsImage.uploadImage(url, conf, storage, nnf, txId, canceler);
} catch (IOException e) { // "Could not find image with txid N"
  // not retryable with the same txid — re-run a full checkpoint first
  triggerCheckpoint();
}

Prevention

Do not run manual fsimage cleanup jobs while checkpoints are in flight
Retain enough checkpoints (dfs.namenode.num.checkpoints.retained) that upload never races deletion
Run exactly one SecondaryNameNode per namespace to avoid concurrent checkpoint/retention churn

When it happens

Trigger: uploadImage(url, conf, storage, nnf, txId, canceler) is called during checkpoint upload right after saving; findImageFile returns null when the image was deleted by retention/cleanup, renamed during a concurrent checkpoint, or when the requested txid never matched a saved file.

Common situations: Image retention purged the just-saved checkpoint between save and upload; an operator or script deleted fsimage files; two checkpointers racing on the same storage; txid arithmetic mismatch between what the peer requested and what was written.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/5acc79452e7df191. Report an issue: GitHub.