apache/hadoop · error · IOException

Missing HTTP 'Location' header for [{0}]

Error message

Missing HTTP 'Location' header for [{0}]

What it means

During WebHDFS/HttpFS create() and append(), the client first receives HTTP 307 Temporary Redirect and then expects a 'Location' header naming the data node (or HttpFS data endpoint) to stream bytes to (HttpFSFileSystem.java:573-594). If the 307 arrives without a Location header, validateResponse runs and this IOException is thrown naming the connection URL. It almost always means something between client and server — a reverse proxy, gateway, or non-WebHDFS service — emitted or rewrote the 307 rather than the HttpFS/WebHDFS endpoint itself.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/client/HttpFSFileSystem.java:590

    conn.setInstanceFollowRedirects(false);
    boolean exceptionAlreadyHandled = false;
    try {
      if (conn.getResponseCode() == HTTP_TEMPORARY_REDIRECT) {
        exceptionAlreadyHandled = true;
        String location = conn.getHeaderField("Location");
        if (location != null) {
          conn = getConnection(new URL(location), method);
          conn.setRequestProperty("Content-Type", UPLOAD_CONTENT_TYPE);
          try {
            OutputStream os = new BufferedOutputStream(conn.getOutputStream(), bufferSize);
            return new HttpFSDataOutputStream(conn, os, expectedStatus, statistics);
          } catch (IOException ex) {
            HttpExceptionUtils.validateResponse(conn, expectedStatus);
            throw ex;
          }
        } else {
          HttpExceptionUtils.validateResponse(conn, HTTP_TEMPORARY_REDIRECT);
          throw new IOException("Missing HTTP 'Location' header for [" + conn.getURL() + "]");
        }
      } else {
        throw new IOException(
          MessageFormat.format("Expected HTTP status was [307], received [{0}]",
                               conn.getResponseCode()));
      }
    } catch (IOException ex) {
      if (exceptionAlreadyHandled) {
        throw ex;
      } else {
        HttpExceptionUtils.validateResponse(conn, HTTP_TEMPORARY_REDIRECT);
        throw ex;
      }
    }
  }


  /**

View on GitHub (pinned to 2add963021)

Solutions

1. Reproduce with curl -i on the exact create/append URL: the 307 must carry a Location header; if absent, the intermediary is the culprit.
2. Fix the proxy: disable/limit redirect rewriting (nginx: proxy_redirect off; do not intercept 3xx), and pass Location through unmodified; or bypass the proxy for the HttpFS data endpoints.
3. If a gateway like Knox is used, upgrade/verify its WebHDFS dispatch so it forwards the datanode/HttpFS Location for the write phase.
4. As a last resort point the client straight at the WebHDFS/HttpFS endpoint (webhdfs://httpfs-host:14000) to confirm the header exists without middleboxes.

Example fix

# before (nginx strips/rewrites the redirect)
location /webhdfs { proxy_pass http://httpfs:14000; proxy_redirect http://https://; }
# after
location /webhdfs { proxy_pass http://httpfs:14000; proxy_redirect off; proxy_set_header Host $host; }
# verify: curl -i -X PUT 'http://gw/webhdfs/v1/f?op=CREATE&noredirect=false' must show Location:

try (FSDataOutputStream out = fs.create(path, true)) {
  out.write(data);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("Missing HTTP 'Location' header")) {
    throw new IOException("WebHDFS/HttpFS 307 redirect lost its Location header — "
        + "check reverse proxy / gateway rewrite rules for " + fs.getUri(), e);
  }
  throw e;
}

When it happens

Trigger: Nginx/haproxy/ALB in front of HttpFS (port 14000) answering or rewriting 307s with proxy_redirect/redirect directives that drop Location; Knox or API-gateway versions that mishandle WebHDFS redirects for the data phase; the create/append URL pointing at a service that returns 307 without Location (e.g. an auth portal redirect captured before status validation); curl-level tests where an HTTP-to-HTTPS redirector strips the header.

Common situations: Corporate TLS terminators proxying webhdfs:// traffic; HttpFS behind Knox with outdated gateway config; switching clients from direct NameNode WebHDFS (9870) to an HttpFS URL and hitting an intermediary for the first time; noredirect=false flows where the data phase still expects the redirect to the HttpFS host itself.

Related errors

• Content-Type "{0}" is incompatible with "application/json" (
• Bad rule definition: {bad_lines}
• Stream closed
• Negative Position
• Content-Length is missing: ${headers}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a82b2eff3da8d90c. Report an issue: GitHub.