{"record":{"id":"075b135a38075c33","repo":"apache/hadoop","slug":"recursive-load-of-credential-provider-if-loading","errorCode":null,"errorMessage":"Recursive load of credential provider; if loading a JCEKS file, this means that the filesystem connector is trying to load the same file","messagePattern":"Recursive load of credential provider; if loading a JCEKS file, this means that the filesystem connector is trying to load the same file","errorType":"exception","errorClass":"PathIOException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/CredentialProviderFactory.java","lineNumber":85,"sourceCode":"   * A simple boolean could be used here, as the synchronized block ensures\n   * that only one thread can be active at a time. An atomic is used\n   * for rigorousness.\n   */\n  private static final AtomicBoolean SERVICE_LOADER_LOCKED = new AtomicBoolean(false);\n\n  public static List<CredentialProvider> getProviders(Configuration conf\n                                               ) throws IOException {\n    List<CredentialProvider> result = new ArrayList<>();\n    for(String path: conf.getStringCollection(CREDENTIAL_PROVIDER_PATH)) {\n      try {\n        URI uri = new URI(path);\n        boolean found = false;\n        // Iterate serviceLoader in a synchronized block since\n        // serviceLoader iterator is not thread-safe.\n        synchronized (serviceLoader) {\n          try {\n            if (SERVICE_LOADER_LOCKED.getAndSet(true)) {\n              throw new PathIOException(path,\n                  \"Recursive load of credential provider; \" +\n                      \"if loading a JCEKS file, this means that the filesystem connector is \" +\n                      \"trying to load the same file\");\n            }\n            for (CredentialProviderFactory factory : serviceLoader) {\n              CredentialProvider kp = factory.createProvider(uri, conf);\n              if (kp != null) {\n                result.add(kp);\n                found = true;\n                break;\n              }\n            }\n          } finally {\n            SERVICE_LOADER_LOCKED.set(false);\n          }\n        }\n        if (!found) {\n          throw new IOException(\"No CredentialProviderFactory for \" + uri + \" in \" +","sourceCodeStart":67,"sourceCodeEnd":103,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/CredentialProviderFactory.java#L67-L103","documentation":"CredentialProviderFactory.getProviders() iterates ServiceLoader-discovered factories under a static SERVICE_LOADER_LOCKED AtomicBoolean; if building a provider re-enters getProviders() on the same thread, it fails fast with this PathIOException. Re-entry happens when resolving the provider URI itself requires loading a FileSystem that, in turn, asks the credential-provider framework for credentials - a bootstrap cycle the code deliberately refuses.","triggerScenarios":"hadoop.security.credential.provider.path contains jceks://hdfs/... while the HDFS client is configured to pull its own secrets from that same provider path; a custom FileSystem/connector whose initialize() calls getProviders(); S3A or other connectors resolving their credential provider from a store that itself lives on the filesystem being mounted.","commonSituations":"Putting the S3A secret store on S3 itself; HDFS encryption/KMS bootstrap credentials stored in HDFS; third-party filesystem implementations that read secrets during connect().","solutions":["Break the cycle: bootstrap secrets from a local store - use localjceks://file/... (or user://) for the first provider entry","Fix the connector's config so it does not resolve credentials from the store it is mounting (e.g. set fs.s3a.security.credential.provider.path to a local store)","If you write a FileSystem, never call CredentialProviderFactory.getProviders() during initialize()/get(); resolve credentials lazily after construction"],"exampleFix":"<!-- before: cycle - store lives on the FS that needs it -->\n<property><name>hadoop.security.credential.provider.path</name>\n  <value>jceks://hdfs/nn1/security/creds.jceks</value></property>\n\n<!-- after: bootstrap locally, keep remote stores in a second layer -->\n<property><name>hadoop.security.credential.provider.path</name>\n  <value>localjceks://file/etc/hadoop/bootstrap.jceks</value></property>","handlingStrategy":"fallback","validationCode":"// Assert the bootstrap provider entries never depend on a remote FileSystem\nstatic void assertBootstrapLocal(Configuration conf) {\n  for (String p : conf.getStringCollection(\"hadoop.security.credential.provider.path\")) {\n    String scheme = URI.create(p).getScheme();\n    if (\"hdfs\".equals(scheme) || scheme == null) {\n      throw new IllegalStateException(\n          \"First credential provider must be local (localjceks:// or user://): \" + p);\n    }\n  }\n}","typeGuard":null,"tryCatchPattern":"try {\n  providers = CredentialProviderFactory.getProviders(conf);\n} catch (org.apache.hadoop.fs.PathIOException ex) {\n  if (ex.getMessage().contains(\"Recursive load\")) {\n    // cycle detected: fall back to a purely local bootstrap store and re-resolve\n    conf.set(\"hadoop.security.credential.provider.path\",\n             \"localjceks://file/etc/hadoop/bootstrap.jceks\");\n    providers = CredentialProviderFactory.getProviders(conf);\n  } else { throw ex; }\n}","preventionTips":["Never place the bootstrap keystore on a filesystem whose client needs those same credentials","Review fs.<scheme>.security.credential.provider.path settings for cycles when adding cloud connectors","In custom FileSystems, load credentials lazily and never from provider construction paths"],"tags":["hadoop","credential-provider","recursive-load","circular-dependency","configuration"],"backgroundTag":"circular-provider-initialization","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}