apache/hadoop · error · PathIOException

Recursive load of credential provider; if loading a JCEKS fi

Error message

Recursive load of credential provider; if loading a JCEKS file, this means that the filesystem connector is trying to load the same file

What it means

CredentialProviderFactory.getProviders() iterates ServiceLoader-discovered factories under a static SERVICE_LOADER_LOCKED AtomicBoolean; if building a provider re-enters getProviders() on the same thread, it fails fast with this PathIOException. Re-entry happens when resolving the provider URI itself requires loading a FileSystem that, in turn, asks the credential-provider framework for credentials - a bootstrap cycle the code deliberately refuses.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/CredentialProviderFactory.java:85

   * A simple boolean could be used here, as the synchronized block ensures
   * that only one thread can be active at a time. An atomic is used
   * for rigorousness.
   */
  private static final AtomicBoolean SERVICE_LOADER_LOCKED = new AtomicBoolean(false);

  public static List<CredentialProvider> getProviders(Configuration conf
                                               ) throws IOException {
    List<CredentialProvider> result = new ArrayList<>();
    for(String path: conf.getStringCollection(CREDENTIAL_PROVIDER_PATH)) {
      try {
        URI uri = new URI(path);
        boolean found = false;
        // Iterate serviceLoader in a synchronized block since
        // serviceLoader iterator is not thread-safe.
        synchronized (serviceLoader) {
          try {
            if (SERVICE_LOADER_LOCKED.getAndSet(true)) {
              throw new PathIOException(path,
                  "Recursive load of credential provider; " +
                      "if loading a JCEKS file, this means that the filesystem connector is " +
                      "trying to load the same file");
            }
            for (CredentialProviderFactory factory : serviceLoader) {
              CredentialProvider kp = factory.createProvider(uri, conf);
              if (kp != null) {
                result.add(kp);
                found = true;
                break;
              }
            }
          } finally {
            SERVICE_LOADER_LOCKED.set(false);
          }
        }
        if (!found) {
          throw new IOException("No CredentialProviderFactory for " + uri + " in " +

View on GitHub (pinned to 2add963021)

Solutions

Break the cycle: bootstrap secrets from a local store - use localjceks://file/... (or user://) for the first provider entry
Fix the connector's config so it does not resolve credentials from the store it is mounting (e.g. set fs.s3a.security.credential.provider.path to a local store)
If you write a FileSystem, never call CredentialProviderFactory.getProviders() during initialize()/get(); resolve credentials lazily after construction

Example fix

<!-- before: cycle - store lives on the FS that needs it -->
<property><name>hadoop.security.credential.provider.path</name>
  <value>jceks://hdfs/nn1/security/creds.jceks</value></property>

<!-- after: bootstrap locally, keep remote stores in a second layer -->
<property><name>hadoop.security.credential.provider.path</name>
  <value>localjceks://file/etc/hadoop/bootstrap.jceks</value></property>

Defensive patterns

Strategy: fallback

Validate before calling

// Assert the bootstrap provider entries never depend on a remote FileSystem
static void assertBootstrapLocal(Configuration conf) {
  for (String p : conf.getStringCollection("hadoop.security.credential.provider.path")) {
    String scheme = URI.create(p).getScheme();
    if ("hdfs".equals(scheme) || scheme == null) {
      throw new IllegalStateException(
          "First credential provider must be local (localjceks:// or user://): " + p);
    }
  }
}

Try / catch

try {
  providers = CredentialProviderFactory.getProviders(conf);
} catch (org.apache.hadoop.fs.PathIOException ex) {
  if (ex.getMessage().contains("Recursive load")) {
    // cycle detected: fall back to a purely local bootstrap store and re-resolve
    conf.set("hadoop.security.credential.provider.path",
             "localjceks://file/etc/hadoop/bootstrap.jceks");
    providers = CredentialProviderFactory.getProviders(conf);
  } else { throw ex; }
}

Prevention

Never place the bootstrap keystore on a filesystem whose client needs those same credentials
Review fs.<scheme>.security.credential.provider.path settings for cycles when adding cloud connectors
In custom FileSystems, load credentials lazily and never from provider construction paths

When it happens

Trigger: hadoop.security.credential.provider.path contains jceks://hdfs/... while the HDFS client is configured to pull its own secrets from that same provider path; a custom FileSystem/connector whose initialize() calls getProviders(); S3A or other connectors resolving their credential provider from a store that itself lives on the filesystem being mounted.

Common situations: Putting the S3A secret store on S3 itself; HDFS encryption/KMS bootstrap credentials stored in HDFS; third-party filesystem implementations that read secrets during connect().

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/075b135a38075c33. Report an issue: GitHub.