{"record":{"id":"eb1331a7fc30a1ed","repo":"apache/hadoop","slug":"could-not-start-secretmanagerclass-ex-tostri","errorCode":null,"errorMessage":"Could not start ${secretManagerClass}: ${ex.toString()}","messagePattern":"Could not start (.+?): (.+?)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"critical","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenManager.java","lineNumber":148,"sourceCode":"   * <p>\n   * This is useful for use cases where secrets must be shared across multiple\n   * services.\n   *\n   * @param secretManager a <code>DelegationTokenSecretManager</code> instance\n   */\n  public void setExternalDelegationTokenSecretManager(\n      AbstractDelegationTokenSecretManager secretManager) {\n    this.secretManager.stopThreads();\n    this.secretManager = secretManager;\n    managedSecretManager = false;\n  }\n\n  public void init() {\n    if (managedSecretManager) {\n      try {\n        secretManager.startThreads();\n      } catch (IOException ex) {\n        throw new RuntimeException(\"Could not start \" +\n            secretManager.getClass() + \": \" + ex.toString(), ex);\n      }\n    }\n  }\n\n  public void destroy() {\n    if (managedSecretManager) {\n      secretManager.stopThreads();\n    }\n  }\n\n  @SuppressWarnings(\"unchecked\")\n  public Token<? extends AbstractDelegationTokenIdentifier> createToken(\n      UserGroupInformation ugi, String renewer) {\n    return createToken(ugi, renewer, null);\n  }\n\n  @SuppressWarnings(\"unchecked\")","sourceCodeStart":130,"sourceCodeEnd":166,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenManager.java#L130-L166","documentation":"DelegationTokenManager.init() starts the wrapped AbstractDelegationTokenSecretManager's threads (startThreads) when it owns the manager (managedSecretManager=true, i.e. no external secret manager was injected via setExternalDelegationTokenSecretManager). For the ZK-backed implementation this connects to ZooKeeper, starts the SharedCounters and creates the persistent roots - any IOException from that whole chain is wrapped as RuntimeException \"Could not start <secretManagerClass>: <msg>\".","triggerScenarios":"Web app initialization (WebHDFS/KMS/HttpFS servlet context, RM workspace) calling tokenManager.init() while ZK is down or auth is broken - the underlying failures are the ZKDTSM start errors (curator start failure, namespace creation, counter start, createPersistentNode).","commonSituations":"Web service starting before ZooKeeper in systemd/K8s ordering; zk-dt-secret-manager.zkConnectionString wrong; Kerberos/JAAS mismatch for secured ZK; failover starting the new active during a quorum partition.","solutions":["Verify ZK from the service host (zkCli four-letter 'stat'/mntr, ls /zkdtsm) and fix connectivity before restarting the web app.","Correct zk-dt-secret-manager.* auth settings (zkAuthType, kerberos.keytab/principal, JAAS) so startThreads can create/start its znodes.","Add startup ordering/dependency (start-after ZK) or a supervised restart so init retries once ZK is healthy.","If you inject your own secret manager, setExternalDelegationTokenSecretManager makes init a no-op for threads - ensure that manager is already started."],"exampleFix":"// before: init blows up as RuntimeException when ZK is briefly down\nDelegationTokenManager tm = new DelegationTokenManager(conf, kind);\ntm.init();\n// after: supervised init with retry once ZK is reachable\nDelegationTokenManager tm = new DelegationTokenManager(conf, kind);\nwhile (true) {\n  try { tm.init(); break; }\n  catch (RuntimeException e) { LOG.warn(\"token manager init failed, retrying\", e); Thread.sleep(5000); }\n}","handlingStrategy":"try-catch","validationCode":"// Before web-app init: confirm the ZK ensemble answers and the working path exists\ntry (CuratorFramework zk = CuratorFrameworkFactory.newClient(quorum, new RetryOneTime(1000))) {\n  zk.start();\n  Preconditions.checkState(zk.checkExists().forPath(\"/zkdtsm\") != null,\n      \"ZK up but delegation-token working path missing\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  tokenManager.init();\n} catch (RuntimeException e) { // \"Could not start <class>: ...\"\n  LOG.error(\"Token manager failed to start; check ZK: {}\", e.getCause().getMessage(), e);\n  // fail the deployment loudly rather than serving an unauthenticated web app\n  throw e;\n}","preventionTips":["Order service startup after ZooKeeper readiness (systemd/K8s dependencies).","Run the ZK connectivity preflight in a health check executed before the web app loads.","If injecting an external secret manager, remember init() then skips thread start - start it yourself."],"tags":["zookeeper","delegation-token","lifecycle","startup"],"backgroundTag":"zookeeper-connection-failure","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}