apache/hadoop · critical · RuntimeException

Could not start ${secretManagerClass}: ${ex.toString()}

Error message

Could not start ${secretManagerClass}: ${ex.toString()}

What it means

DelegationTokenManager.init() starts the wrapped AbstractDelegationTokenSecretManager's threads (startThreads) when it owns the manager (managedSecretManager=true, i.e. no external secret manager was injected via setExternalDelegationTokenSecretManager). For the ZK-backed implementation this connects to ZooKeeper, starts the SharedCounters and creates the persistent roots - any IOException from that whole chain is wrapped as RuntimeException "Could not start <secretManagerClass>: <msg>".

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenManager.java:148

   * <p>
   * This is useful for use cases where secrets must be shared across multiple
   * services.
   *
   * @param secretManager a <code>DelegationTokenSecretManager</code> instance
   */
  public void setExternalDelegationTokenSecretManager(
      AbstractDelegationTokenSecretManager secretManager) {
    this.secretManager.stopThreads();
    this.secretManager = secretManager;
    managedSecretManager = false;
  }

  public void init() {
    if (managedSecretManager) {
      try {
        secretManager.startThreads();
      } catch (IOException ex) {
        throw new RuntimeException("Could not start " +
            secretManager.getClass() + ": " + ex.toString(), ex);
      }
    }
  }

  public void destroy() {
    if (managedSecretManager) {
      secretManager.stopThreads();
    }
  }

  @SuppressWarnings("unchecked")
  public Token<? extends AbstractDelegationTokenIdentifier> createToken(
      UserGroupInformation ugi, String renewer) {
    return createToken(ugi, renewer, null);
  }

  @SuppressWarnings("unchecked")

View on GitHub (pinned to 2add963021)

Solutions

1. Verify ZK from the service host (zkCli four-letter 'stat'/mntr, ls /zkdtsm) and fix connectivity before restarting the web app.
2. Correct zk-dt-secret-manager.* auth settings (zkAuthType, kerberos.keytab/principal, JAAS) so startThreads can create/start its znodes.
3. Add startup ordering/dependency (start-after ZK) or a supervised restart so init retries once ZK is healthy.
4. If you inject your own secret manager, setExternalDelegationTokenSecretManager makes init a no-op for threads - ensure that manager is already started.

Example fix

// before: init blows up as RuntimeException when ZK is briefly down
DelegationTokenManager tm = new DelegationTokenManager(conf, kind);
tm.init();
// after: supervised init with retry once ZK is reachable
DelegationTokenManager tm = new DelegationTokenManager(conf, kind);
while (true) {
  try { tm.init(); break; }
  catch (RuntimeException e) { LOG.warn("token manager init failed, retrying", e); Thread.sleep(5000); }
}

// Before web-app init: confirm the ZK ensemble answers and the working path exists
try (CuratorFramework zk = CuratorFrameworkFactory.newClient(quorum, new RetryOneTime(1000))) {
  zk.start();
  Preconditions.checkState(zk.checkExists().forPath("/zkdtsm") != null,
      "ZK up but delegation-token working path missing");
}

try {
  tokenManager.init();
} catch (RuntimeException e) { // "Could not start <class>: ..."
  LOG.error("Token manager failed to start; check ZK: {}", e.getCause().getMessage(), e);
  // fail the deployment loudly rather than serving an unauthenticated web app
  throw e;
}

When it happens

Trigger: Web app initialization (WebHDFS/KMS/HttpFS servlet context, RM workspace) calling tokenManager.init() while ZK is down or auth is broken - the underlying failures are the ZKDTSM start errors (curator start failure, namespace creation, counter start, createPersistentNode).

Common situations: Web service starting before ZooKeeper in systemd/K8s ordering; zk-dt-secret-manager.zkConnectionString wrong; Kerberos/JAAS mismatch for secured ZK; failover starting the new active during a quorum partition.

Related errors

• ${nodePath} znode could not be created !!
• Can't update persisted delegation token renewal to a running
• signer.secret.provider.zookeeper.path must be specified
• Could not increment shared counter !!
• Could not set shared counter !!

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/eb1331a7fc30a1ed. Report an issue: GitHub.