apache/hadoop · error · HttpPutFailedException

Image uploading failed, status: %d, url: %s, message: %s

Error message

Image uploading failed, status: %d, url: %s, message: %s

What it means

TransferFsImage.uploadImage() streams the image with HTTP PUT and checks connection.getResponseCode(); any non-200 answer raises HttpPutFailedException carrying the status code, the full URL, and the server's response message. The actionable cause is in the peer's status/message and the peer NameNode's log.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java:318

      if (imageFile.length() > chunkSize) {
        // using chunked streaming mode to support upload of 2GB+ files and to
        // avoid internal buffering.
        // this mode should be used only if more than chunkSize data is present
        // to upload. otherwise upload may not happen sometimes.
        connection.setChunkedStreamingMode(chunkSize);
      }

      setTimeout(connection);

      // set headers for verification
      ImageServlet.setVerificationHeadersForPut(connection, imageFile);

      // Write the file to output stream.
      writeFileToPutRequest(conf, connection, imageFile, canceler, chunkSize);

      int responseCode = connection.getResponseCode();
      if (responseCode != HttpURLConnection.HTTP_OK) {
        throw new HttpPutFailedException(String.format(
            "Image uploading failed, status: %d, url: %s, message: %s",
            responseCode, urlWithParams, connection.getResponseMessage()),
            responseCode);
      }
    } catch (AuthenticationException | URISyntaxException e) {
      throw new IOException(e);
    } finally {
      if (connection != null) {
        connection.disconnect();
      }
    }
  }

  private static void writeFileToPutRequest(Configuration conf,
      HttpURLConnection connection, File imageFile, Canceler canceler,
      int bufferSize)
      throws IOException {
    connection.setRequestProperty(Util.CONTENT_TYPE, "application/octet-stream");

View on GitHub (pinned to 2add963021)

Solutions

1. Read the embedded status and message, then check the peer NameNode log at the same timestamp for the server-side reason
2. For 401/403: fix Kerberos (kinit/keytab, SPNEGO principal, HTTP policy) between the two nodes
3. For 5xx: check the peer's checkpoint directory space and writability
4. Ensure a single checkpointer per namespace and correct http-address/http-policy in config, then let the next checkpoint retry

int attempts = 0;
while (true) {
  try {
    TransferFsImage.uploadImage(url, conf, storage, nnf, txId, canceler);
    break;
  } catch (TransferFsImage.HttpPutFailedException e) {
    int code = e.getStatusCode();
    if (code == 401 || code == 403) throw e;       // auth: fix Kerberos, do not retry
    if (++attempts >= 3) throw e;                   // give up after 3
    Thread.sleep(backoffMs << attempts);            // 5xx/timeout: retry with backoff
  }
}

When it happens

Trigger: PUT to the peer's ImageServlet (putImage endpoint) returns e.g. 401/403 on Kerberos/SPNEGO failure, 403/409 when the peer rejects the txid or another transfer is in progress, 500 on peer-side disk/IO failure, or a proxy/timeout returns 4xx/5xx because of a wrong http-address or http-policy mismatch.

Common situations: Expired or missing SPNEGO credentials between 2NN and NN; peer's checkpoint volume full; concurrent putImage from two secondaries; SSL/http vs https misconfiguration of dfs.namenode.http(s)-address; an intermediate proxy stripping PUT or auth headers.

Related errors

• Image transfer servlet at " + url + " failed with status cod
• Could not find image with txid ${txId}
• The length of the feature flag section was negative at {} by
• Security is enabled but block access tokens (via dfs.block.a
• Content-Length header is not provided by the namenode when t

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/b4ba9a4de4743388. Report an issue: GitHub.