apache/hadoop · error · IOException

Unexpected HTTP response: code=${code} != ${op.getExpectedHt

Error message

Unexpected HTTP response: code=${code} != ${op.getExpectedHttpResponseCode()}, ${op.toQueryString()}, message=${conn.getResponseMessage()}

What it means

validateResponse checks every WebHDFS operation's HTTP status against the expected code. When the status differs and the error body cannot be parsed as JSON, it throws this IOException containing the actual code, expected code, operation query string, and HTTP message. The JSON parse failure is attached as the cause, which usually identifies an HTML/plain-text error body or an unavailable stream.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java:524

      in.close();
    }
  }

  private static Map<?, ?> validateResponse(final HttpOpParam.Op op,
      final HttpURLConnection conn, boolean unwrapException)
      throws IOException {
    final int code = conn.getResponseCode();
    // server is demanding an authentication we don't support
    if (code == HttpURLConnection.HTTP_UNAUTHORIZED) {
      // match hdfs/rpc exception
      throw new AccessControlException(conn.getResponseMessage());
    }
    if (code != op.getExpectedHttpResponseCode()) {
      final Map<?, ?> m;
      try {
        m = jsonParse(conn, true);
      } catch(Exception e) {
        throw new IOException("Unexpected HTTP response: code=" + code + " != "
            + op.getExpectedHttpResponseCode() + ", " + op.toQueryString()
            + ", message=" + conn.getResponseMessage(), e);
      }

      if (m == null) {
        throw new IOException("Unexpected HTTP response: code=" + code + " != "
            + op.getExpectedHttpResponseCode() + ", " + op.toQueryString()
            + ", message=" + conn.getResponseMessage());
      } else if (m.get(RemoteException.class.getSimpleName()) == null) {
        return m;
      }

      IOException re = JsonUtilClient.toRemoteException(m);

      //check if exception is due to communication with a Standby name node
      if (re.getMessage() != null && re.getMessage().endsWith(
          StandbyException.class.getSimpleName())) {
        LOG.trace("Detected StandbyException", re);

View on GitHub (pinned to 2add963021)

Solutions

1. Replay the operation with curl -i using the same URL, query string, and credentials, and inspect the status plus body shown in the exception.
2. Check NameNode/HttpFS and proxy logs for the corresponding request to find the real server-side cause.
3. Correct endpoint URLs, ports, authentication, and CSRF configuration if the response body shows another service or filter answered.
4. Retry only after confirming a transient server condition, because configuration and routing failures will reproduce identically.

Example fix

# before: exception shows code=502 != 200, op=OPEN
# typical reverse-proxy HTML error is the cause

# after: inspect and bypass the failing hop
curl -i --negotiate -u : "http://nn:9870/webhdfs/v1/user/alice/f?op=OPEN&offset=0&length=1024"
# then configure the client to call nn:9870 directly rather than the proxy returning 502

try {
  return fs.open(path);
} catch (IOException e) {
  String message = String.valueOf(e.getMessage());
  if (message.startsWith("Unexpected HTTP response: code=")) {
    // Configuration/routing failures are not retryable; transient 5xx from an active NN may be.
    LOG.error("WebHDFS returned an unexpected non-JSON response: {}", message, e);
  }
  throw e;
}

When it happens

Trigger: A WebHDFS operation receives an unexpected HTTP status and jsonParse(conn, true) throws while reading its error stream. Examples are a proxy returning an HTML 502 page, a server error page, a request routed to a non-WebHDFS service, or a truncated error response. HTTP 401 is handled separately as AccessControlException.

Common situations: NameNode or HttpFS outage behind a gateway; wrong URL or port; firewall/proxy 5xx pages; CSRF or authentication filters issuing non-JSON responses; disk, memory, or handler failures on the server.

Related errors

• The ${useErrorStream ? "error" : "input"} stream is null.
• Content-Type "${contentType}" is incompatible with "applicat
• Missing response
• Invalid value in server response: name=[${name}]
• Missing both 'ipAddr' and 'name' in server response.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6e1bfe17becd999a. Report an issue: GitHub.