apache/hadoop · error · IllegalArgumentException

url must be for a HTTP or HTTPS resource

Error message

url must be for a HTTP or HTTPS resource

What it means

AuthenticatedURL only speaks plain HTTP/S: after the null check it requires the URL protocol to be http or https (case-insensitive) and otherwise throws IllegalArgumentException('url must be for a HTTP or HTTPS resource'). This is because the class authenticates over HttpURLConnection cookies/SPNEGO at the HTTP layer, not over Hadoop RPC or other schemes. Passing hdfs://, webhdfs:// (the REST scheme), ftp://, or file:// URLs hits this guard.

Source

Thrown at hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/AuthenticatedURL.java:348

  }

  /**
   * Returns an authenticated {@link HttpURLConnection}.
   *
   * @param url the URL to connect to. Only HTTP/S URLs are supported.
   * @param token the authentication token being used for the user.
   *
   * @return an authenticated {@link HttpURLConnection}.
   *
   * @throws IOException if an IO error occurred.
   * @throws AuthenticationException if an authentication exception occurred.
   */
  public HttpURLConnection openConnection(URL url, Token token) throws IOException, AuthenticationException {
    if (url == null) {
      throw new IllegalArgumentException("url cannot be NULL");
    }
    if (!url.getProtocol().equalsIgnoreCase("http") && !url.getProtocol().equalsIgnoreCase("https")) {
      throw new IllegalArgumentException("url must be for a HTTP or HTTPS resource");
    }
    if (token == null) {
      throw new IllegalArgumentException("token cannot be NULL");
    }
    authenticator.authenticate(url, token);

    // allow the token to create the connection with a cookie handler for
    // managing session cookies.
    return token.openConnection(url, connConfigurator);
  }

  /**
   * Helper method that injects an authentication token to send with a
   * connection. Callers should prefer using
   * {@link Token#openConnection(URL, ConnectionConfigurator)} which
   * automatically manages authentication tokens.
   *
   * @param conn connection to inject the authentication token into.

View on GitHub (pinned to 2add963021)

Solutions

1. Use the HTTP endpoint: http://host:port/... — for WebHDFS REST, http://nn:9870/webhdfs/v1<path>?op=... (or the https port with SSL enabled).
2. Convert scheme programmatically when reusing a WebHDFS URI: new URI('http', uri.getUserInfo(), uri.getHost(), port, uri.getPath(), ...) or simple scheme replacement webhdfs->http / swebhdfs->https.
3. Check the service's documented HTTP port (NameNode UI / httpfs / ATS) rather than reusing the RPC port.
4. Add a startup assertion on url.getProtocol() to fail with your own message naming the bad value.

Example fix

// before
URL u = new URL('webhdfs://nn:9870/webhdfs/v1/data?f=1'); // non-HTTP scheme
new AuthenticatedURL().openConnection(u, token);

// after
URL u = new URL('http://nn:9870/webhdfs/v1/data?op=OPEN&...');
new AuthenticatedURL().openConnection(u, token);

String proto = url.getProtocol();
if (!("http".equalsIgnoreCase(proto) || "https".equalsIgnoreCase(proto))) {
  throw new IllegalArgumentException("AuthenticatedURL needs http/https, got: " + proto);
}

static boolean isHttpUrl(URL u) {
  String p = u.getProtocol();
  return "http".equalsIgnoreCase(p) || "https".equalsIgnoreCase(p);
}

When it happens

Trigger: Constructing a URL from an HDFS URI (hdfs://nn:8020/path) instead of the service's HTTP endpoint (http://nn:9874/... or the WebHDFS http endpoint http://nn:9870/webhdfs/v1/...); parsing a string like webhdfs://host:9870/... that was never converted to http; hardcoded ftp:// or file:// test URLs.

Common situations: Copying filesystem URIs from core-site fs.defaultFS into client code that talks to the NameNode/JobHistory/ATS web endpoints; tools that accept one URI and must talk to both RPC and HTTP planes; new WebHDFS users unaware the http(s) scheme is required for the REST layer.

Related errors

• url cannot be NULL
• tokenStr cannot be null
• token cannot be NULL
• Authentication failed, URL: {}, status: {}, message: {}
• Invalid SPNEGO sequence, 'WWW-Authenticate' header incorrect

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ad938247cdf2bfca. Report an issue: GitHub.