What causes "Cannot find password option {}" in apache/hadoop?

Validate the provider chain: hadoop credential list -provider jceks://file/... and confirm the store file exists, is readable by the running user on EVERY node.

apache/hadoop · error · IOException

Cannot find password option {}

Error message

Cannot find password option {}

What it means

OBSCommonUtils.lookupPassword(conf, key) calls conf.getPassword(key) — the Hadoop credential-provider-aware accessor — and wraps any IOException it throws as IOException('Cannot find password option <key>'). The underlying failure is almost never a missing value (getPassword returns null for that); it is a broken credential provider configuration, e.g. a malformed or unreadable provider in hadoop.security.credential.provider.path (or fs.obs.security.credential.provider.path) that throws when the password needs to be resolved.

Source

Thrown at hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/java/org/apache/hadoop/fs/obs/OBSCommonUtils.java:1227

      final String key, final String val) throws IOException {
    return StringUtils.isEmpty(val) ? lookupPassword(conf, key) : val;
  }

  /**
   * Get a password from a configuration/configured credential providers.
   *
   * @param conf configuration
   * @param key  key to look up
   * @return a password or the value in {@code defVal}
   * @throws IOException on any problem
   */
  private static String lookupPassword(final Configuration conf,
      final String key) throws IOException {
    try {
      final char[] pass = conf.getPassword(key);
      return pass != null ? new String(pass).trim() : "";
    } catch (IOException ioe) {
      throw new IOException("Cannot find password option " + key, ioe);
    }
  }

  /**
   * String information about a summary entry for debug messages.
   *
   * @param summary summary object
   * @return string value
   */
  static String stringify(final ObsObject summary) {
    return summary.getObjectKey() + " size=" + summary.getMetadata()
        .getContentLength();
  }

  /**
   * Get a integer option not smaller than the minimum allowed value.
   *
   * @param conf   configuration

View on GitHub (pinned to 2add963021)

Solutions

Validate the provider chain: hadoop credential list -provider jceks://file/... and confirm the store file exists, is readable by the running user on EVERY node.
Re-create the entry: hadoop credential create fs.obs.secret.key -provider jceks://file/user/... and re-distribute with correct ownership/permissions.
Check the alias matches the option key exactly (fs.obs.secret.key; per-bucket variants use the bucket-qualified name).
If you do not use credential providers, ensure the provider path config is empty/valid and supply AK/SK via fs.obs.access.key / fs.obs.secret.key or env vars so getPassword never consults a broken provider.

Example fix

# before
# hadoop.security.credential.provider.path = jceks://file/opt/secrets/obs.jceks
# (file missing on workers) -> Cannot find password option fs.obs.secret.key

# after
hadoop credential create fs.obs.secret.key \
  -provider jceks://file/opt/secrets/obs.jceks
# distribute with correct perms:
#   chown hdfs:hdfs /opt/secrets/obs.jceks; chmod 440 /opt/secrets/obs.jceks

Defensive patterns

Strategy: validation

Validate before calling

static void verifyCredentialProvider(Configuration conf) throws IOException {
  String path = conf.get("hadoop.security.credential.provider.path", "");
  for (String entry : path.split(",")) {
    if (entry.startsWith("jceks://file")) {
      java.io.File f = new java.io.File(entry.substring("jceks://file".length()));
      if (!f.canRead()) throw new IOException("unreadable credential store: " + f);
    }
  }
}

Try / catch

try {
  obsFs.initialize(uri, conf);
} catch (IOException e) {
  if (String.valueOf(e.getMessage()).startsWith("Cannot find password option")) {
    throw new ConfigException("credential provider chain broken — check provider path/perms/alias", e);
  }
  throw e;
}

Prevention

Run 'hadoop credential list -provider ...' as a cluster bootstrap check.
Distribute .jceks files with strict ownership (running user) and mode 440/400.
Keep alias names exactly equal to the option keys; add a config-lint that cross-checks aliases.
Do not confuse: a missing value returns null (no throw); a throwing provider is a broken provider.

When it happens

Trigger: hadoop.security.credential.provider.path contains a jceks:// URL whose file does not exist or has wrong permissions; a localjceks/JavaKeyStoreProvider with a corrupt store; an HadoopKMS provider unreachable/misconfigured; password stored under a different alias than the key being looked up so a failing provider still gets consulted.

Common situations: Migrating secrets to credential providers at upgrade time; cluster nodes where the .jceks file was distributed with restrictive ownership (only deployed user can read); alias typos like fs.obs.secet.key; mixing per-bucket fs.obs.bucket.X.security.credential.provider.path overrides pointing at stale locations.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e8fa8871b63ecd73. Report an issue: GitHub.