apache/hadoop · error · IllegalArgumentException

Proxy error: fs.s3a.proxy.username or fs.s3a.proxy.password

Error message

Proxy error: fs.s3a.proxy.username or fs.s3a.proxy.password set without the other.

What it means

IllegalArgumentException from AWSClientConfig's sync proxy setup when exactly one of fs.s3a.proxy.username / fs.s3a.proxy.password is configured ((username == null) != (password == null)). The AWS SDK proxy builder requires both credentials together. Passwords are resolved through S3AUtils.lookupPassword, so the password may come from a credential provider file -- if that entry is missing on some node, the value resolves to null there and triggers this error even though the username is set in core-site.xml.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/AWSClientConfig.java:260

        proxyConfigBuilder.endpoint(buildURI(scheme, proxyHost, proxyPort));
      } else {
        if (conf.getBoolean(PROXY_SECURED, false)) {
          LOG.warn("Proxy host set without port. Using HTTPS default 443");
          proxyConfigBuilder.endpoint(buildURI("https", proxyHost, 443));
        } else {
          LOG.warn("Proxy host set without port. Using HTTP default 80");
          proxyConfigBuilder.endpoint(buildURI("http", proxyHost, 80));
        }
      }
      final String proxyUsername = S3AUtils.lookupPassword(bucket, conf, PROXY_USERNAME,
          null, null);
      final String proxyPassword = S3AUtils.lookupPassword(bucket, conf, PROXY_PASSWORD,
          null, null);
      if ((proxyUsername == null) != (proxyPassword == null)) {
        String msg = "Proxy error: " + PROXY_USERNAME + " or " +
            PROXY_PASSWORD + " set without the other.";
        LOG.error(msg);
        throw new IllegalArgumentException(msg);
      }
      proxyConfigBuilder.username(proxyUsername);
      proxyConfigBuilder.password(proxyPassword);
      proxyConfigBuilder.ntlmDomain(conf.getTrimmed(PROXY_DOMAIN));
      proxyConfigBuilder.ntlmWorkstation(conf.getTrimmed(PROXY_WORKSTATION));
      if (LOG.isDebugEnabled()) {
        LOG.debug("Using proxy server {}:{} as user {} with password {} on "
                + "domain {} as workstation {}", proxyHost, proxyPort, proxyUsername, proxyPassword,
            PROXY_DOMAIN, PROXY_WORKSTATION);
      }
    } else if (proxyPort >= 0) {
      String msg =
          "Proxy error: " + PROXY_PORT + " set without " + PROXY_HOST;
      LOG.error(msg);
      throw new IllegalArgumentException(msg);
    }

    return proxyConfigBuilder.build();

View on GitHub (pinned to 2add963021)

Solutions

Set both fs.s3a.proxy.username and fs.s3a.proxy.password (or remove both if the proxy needs no authentication)
If the password comes from a credential provider, verify the provider file exists on every node and actually contains the fs.s3a.proxy.password entry
Note the same rule is enforced in the async client path (line ~329), so fixing only one code path is not enough -- fix the config

Example fix

<!-- before -->
<property><name>fs.s3a.proxy.host</name><value>proxy.corp</value></property>
<property><name>fs.s3a.proxy.username</name><value>alice</value></property>

<!-- after -->
<property><name>fs.s3a.proxy.host</name><value>proxy.corp</value></property>
<property><name>fs.s3a.proxy.username</name><value>alice</value></property>
<property><name>fs.s3a.proxy.password</name><value>${credential-store alias}</value></property>

Defensive patterns

Strategy: validation

Validate before calling

Configuration c = fs.getConf();
String u = S3AUtils.lookupPassword(bucket, c, PROXY_USERNAME, null, null);
String p = S3AUtils.lookupPassword(bucket, c, PROXY_PASSWORD, null, null);
if ((u == null) != (p == null)) {
  throw new IllegalArgumentException("Both proxy username and password must be set, or neither");
}

Try / catch

try {
  FileSystem fs = path.getFileSystem(conf);
} catch (IllegalArgumentException e) {
  if (e.getMessage().contains("fs.s3a.proxy.username")) {
    // config problem, not code: fix core-site/credential provider, no point retrying
    throw new ConfigurationException("Proxy credentials incomplete", e);
  }
  throw e;
}

Prevention

Deploy credential-provider files containing fs.s3a.proxy.password to every node, not just gateways
Automate config checks that assert username and password are both present or both absent
Remember lookupPassword resolves from providers -- a missing alias behaves like an unset key

When it happens

Trigger: Building the S3A client (any FS initialization, e.g. FileSystem.get on s3a://) on a node where fs.s3a.proxy.username is set but fs.s3a.proxy.password is unset, or vice versa.

Common situations: Username hardcoded in core-site.xml while the password is sourced from a password file/jceks that is missing or lacks the fs.s3a.proxy.password entry on part of the cluster; password was removed for a 'no-auth proxy' migration but the username was forgotten.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a29efcc9ce482fd6. Report an issue: GitHub.