apache/hadoop · critical · IOException

Access denied: dfs.http.policy is HTTPS_ONLY.

Error message

Access denied: dfs.http.policy is HTTPS_ONLY.

What it means

Before connecting, WebHdfsFileSystem checks whether the target URL uses plain http while Kerberos security is enabled and the client configuration says dfs.http.policy is HTTPS_ONLY. It deliberately refuses the connection so Kerberos-authenticated WebHDFS traffic is never downgraded to plaintext HTTP. This is a client-side policy enforcement check, not a response from the server.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java:729

     * send out the data before the redirect. This issue is addressed by the
     * "Expect: 100-continue" header in HTTP/1.1; see RFC 2616, Section 8.2.3.
     * Unfortunately, there are software library bugs (e.g. Jetty 6 http server
     * and Java 6 http client), which do not correctly implement "Expect:
     * 100-continue". The two-step create/append is a temporary workaround for
     * the software library bugs.
     *
     * Open/Checksum
     * Also implements two-step connects for other operations redirected to
     * a DN such as open and checksum
     */
    protected HttpURLConnection connect(URL url) throws IOException {
      //redirect hostname and port
      redirectHost = null;

      if (url.getProtocol().equals("http") &&
        UserGroupInformation.isSecurityEnabled() &&
        isTLSKrb) {
        throw new IOException("Access denied: dfs.http.policy is HTTPS_ONLY.");
      }

      // resolve redirects for a DN operation unless already resolved
      if (op.getRedirect() && !redirected) {
        final HttpOpParam.Op redirectOp =
            HttpOpParam.TemporaryRedirectOp.valueOf(op);
        final HttpURLConnection conn = connect(redirectOp, url);
        // application level proxy like httpfs might not issue a redirect
        if (conn.getResponseCode() == op.getExpectedHttpResponseCode()) {
          return conn;
        }
        try {
          validateResponse(redirectOp, conn, false);
          url = new URL(conn.getHeaderField("Location"));
          redirectHost = url.getHost() + ":" + url.getPort();
        } finally {
          // TODO: consider not calling conn.disconnect() to allow connection reuse
          // See http://tinyurl.com/java7-http-keepalive

View on GitHub (pinned to 2add963021)

Solutions

1. Change the FileSystem URI from webhdfs://namenode:http-port to swebhdfs://namenode:https-port.
2. Ensure the client can trust the server certificate and has valid hadoop.ssl.client.conf / truststore settings.
3. Only if the security policy intentionally allows HTTP, change dfs.http.policy on the server and client to HTTP_AND_HTTPS or HTTP_ONLY; do not weaken HTTPS_ONLY merely to bypass the error.
4. Audit configuration and URLs together during a security-policy migration so scheme, port, and dfs.http.policy agree.

Example fix

// before
FileSystem fs = FileSystem.get(new URI("webhdfs://nn:9870"), conf);

// after
FileSystem fs = FileSystem.get(new URI("swebhdfs://nn:9871"), conf);

if (UserGroupInformation.isSecurityEnabled()
    && "HTTPS_ONLY".equals(conf.get("dfs.http.policy"))
    && "webhdfs".equalsIgnoreCase(new Path(String.valueOf(url)).toUri().getScheme())) {
  throw new IllegalArgumentException("Use swebhdfs:// because dfs.http.policy is HTTPS_ONLY");
}

try {
  return FileSystem.get(new URI("webhdfs://nn:9870"), conf);
} catch (IOException e) {
  if ("Access denied: dfs.http.policy is HTTPS_ONLY.".equals(e.getMessage())) {
    return FileSystem.get(new URI("swebhdfs://nn:9871"), conf);
  }
  throw e;
}

When it happens

Trigger: Initializing or using a FileSystem with a webhdfs:// URL when UserGroupInformation.isSecurityEnabled() is true and dfs.http.policy=HTTPS_ONLY in the client Configuration. The equivalent HTTPS URL with swebhdfs:// does not trigger it.

Common situations: A cluster is configured for HTTPS_ONLY but jobs or documentation still use webhdfs://; port copied from the HTTP endpoint; a security hardening change was not followed by client URL updates; client core-site.xml retains the server's HTTPS_ONLY policy while the code chooses http.

Understand the failure class

• Authentication and authorization failures — expired tokens, bad credentials, and missing scopes.

Related errors

• Server asks us to fall back to SIMPLE auth, but this client
• Security is enabled but block access tokens (via dfs.block.a
• Failed to obtain current username
• Unable to load OAuth2 connection factory.
• Can't find KeyProvider for key ${keyName}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/22cc01ce9132ea53. Report an issue: GitHub.