apache/hadoop · error · IOException

PutImage failed. " + StringUtils.stringifyException(t)

Error message

PutImage failed. " + StringUtils.stringifyException(t)

What it means

The PUT counterpart of GetImage: ImageServlet.putImage wraps every Throwable raised while the NameNode receives and persists a checkpoint fsimage uploaded by the Secondary - it streams the image into FSImage storage inside a fenced callable, checks the announced size/txid and updates checkpointing state. On failure the peer gets HTTP 410 Gone with the stringified cause and the servlet rethrows IOException. As with GetImage, the actionable cause is the nested exception, not this wrapper.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ImageServlet.java:700

                  nnImage.purgeOldStorage(nnf);
                } finally {
                  // remove the request once we've processed it, or it threw an error, so we
                  // aren't using it either
                  currentlyDownloadingCheckpoints.remove(imageRequest);

                  stream.close();
                }
              } finally {
                nnImage.removeFromCheckpointing(txid);
              }
              return null;
            }

          });
    } catch (Throwable t) {
      String errMsg = "PutImage failed. " + StringUtils.stringifyException(t);
      sendError(response, HttpServletResponse.SC_GONE, errMsg);
      throw new IOException(errMsg);
    }
  }

  private void sendError(HttpServletResponse response, int code, String message)
      throws IOException {
    if (response instanceof Response) {
      ((Response)response).setStatusWithReason(code, message);
    }

    response.sendError(code, message);
  }

  /*
   * Params required to handle put image request
   */
  static class PutImageParams {
    private long txId = -1;
    private String storageInfoString = null;

View on GitHub (pinned to 2add963021)

Solutions

Unwrap the nested stack in the 410 body / NN log - root causes range from DiskErrorException to EOFException on truncated upload.
Verify free space and write permission on every dfs.namenode.name.dir of the receiving NameNode.
On MD5/length mismatch, simply retry the checkpoint; if uploads keep truncating, inspect MTU, proxies and SPNEGO on the HTTP channel to port 9870.
Confirm the 2NN is checkpointing against the right namespace - a storage-info mismatch often surfaces here as a PutImage failure.

Defensive patterns

Strategy: try-catch

Validate before calling

// Uploader side: verify the announced metadata before the PUT
if (imageFile.length() <= 0 || txid < 0 || storageInfo == null || storageInfo.isEmpty()) {
  throw new IllegalStateException("Refusing PUT: fileLength/txid/storageInfo incomplete");
}

Try / catch

try {
  secondary.uploadImageToActive(txid, imageFile); // 2NN push path
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("PutImage failed")) {
    LOG.warn("Upload rejected; root cause is nested - aborting this checkpoint cycle");
    scheduler.scheduleNextCheckpoint(); // fresh attempt with new txid
    return;
  }
  throw e;
}

Prevention

Keep headroom on every dfs.namenode.name.dir; disk-full is the most common root cause.
Align NN and 2NN versions so upload metadata (fileLength, imageType) matches the contract.
Watch the checkpoint metrics (LastCheckpointTime, transactions since checkpoint) to detect persistently failing uploads early.

When it happens

Trigger: 2NN uploads the image (PUT with txid/fileLength/storageinfo) and the NN fails to save it: disk full or name dir removed, MD5/size mismatch against the announced fileLength, the Checkpointer rejecting a stale txid, or an exception in the checkpoint-state management around nnImage.removeFromCheckpointing.

Common situations: Checkpoint name dirs full or read-only; NN storage reformatted while the 2NN kept old state; truncated uploads through proxies or flaky Kerberos HTTP channels; concurrent checkpoints from HA pairs racing the upload.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/5e8b7ca625ac0b18. Report an issue: GitHub.