apache/hadoop · critical · IOException

Failed to create SecretManager

Error message

Failed to create SecretManager

What it means

RouterSecurityManager is created when hadoop.security.authentication=kerberos; it instantiates the Router's DelegationTokenSecretManager via FederationUtil.newSecretManager using the class in dfs.federation.router.delegation-token.driver-class. If the returned instance is null (class could not be instantiated) or isRunning() is false (its threads were never started / startThreads failed), the manager cannot issue tokens for the federation, so the constructor throws this IOException and the Router fails to start.

Solutions

1. Read the log lines immediately before this exception — FederationUtil.newSecretManager and the driver implementation log the underlying instantiation/startThreads failure (SQL connect error, ZK connect error, ClassNotFoundException).
2. Fix the backend the token driver depends on: verify MySQL connectivity/credentials/schema or ZooKeeper reachability from the router host.
3. Verify dfs.federation.router.delegation-token.driver-class names a class on the router classpath that extends AbstractDelegationTokenSecretManager.
4. If you do not need Router-issued delegation tokens, leave the default driver and ensure kerberos configuration is otherwise complete.
5. Restart the Router once the backend is healthy — this check runs only at startup.

// Router startup / tooling that instantiates the security manager:
try {
  RouterSecurityManager sm = new RouterSecurityManager(conf);
} catch (IOException e) {
  if ("Failed to create SecretManager".equals(e.getMessage())) {
    // token driver backend (MySQL/ZK) or driver class is broken: fail startup loudly,
    // check earlier log lines for the underlying instantiation/startThreads error
    throw new IllegalStateException("Router token secret manager unavailable; check "
        + "dfs.federation.router.delegation-token.driver-class backend", e);
  }
  throw e;
}

When it happens

Trigger: Instantiating RouterSecurityManager with kerberos enabled while the configured token driver class is missing, mistyped, not an AbstractDelegationTokenSecretManager, or throws in its constructor; or the driver (SQL/ZooKeeper-backed impl) constructs but startThreads fails because its backend (MySQL, ZooKeeper) is unreachable — its own wrapper exception surfaces right before this one.

Common situations: Switching dfs.federation.router.delegation-token.driver-class to SQLDelegationTokenSecretManagerImpl or ZKDelegationTokenSecretManagerImpl without provisioning the MySQL schema or ZK ensemble; wrong JDBC credentials/URL; ZK connection string wrong; class not on the router classpath after a custom build; MySQL down at router restart with the SQL driver configured.

Related errors

• Delegation Token can be issued only with kerberos or web…
• Delegation Token can be renewed only with kerberos or web…
• Error starting threads for MySQL secret manager
• Fetch of delegation token failed
• Security is enabled but block access tokens (via…

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/20b1ffe2676fc5a9. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/security/RouterSecurityManager.java:62

 * Manager to hold underlying delegation token secret manager implementations.
 */
public class RouterSecurityManager {

  private static final Logger LOG =
      LoggerFactory.getLogger(RouterSecurityManager.class);

  private AbstractDelegationTokenSecretManager<DelegationTokenIdentifier>
      dtSecretManager = null;

  public RouterSecurityManager(Configuration conf) throws IOException {
    AuthenticationMethod authMethodConfigured =
        SecurityUtil.getAuthenticationMethod(conf);
    AuthenticationMethod authMethodToInit =
        AuthenticationMethod.KERBEROS;
    if (authMethodConfigured.equals(authMethodToInit)) {
      this.dtSecretManager = FederationUtil.newSecretManager(conf);
      if (this.dtSecretManager == null || !this.dtSecretManager.isRunning()) {
        throw new IOException("Failed to create SecretManager");
      }
    }
  }

  @VisibleForTesting
  public RouterSecurityManager(AbstractDelegationTokenSecretManager
      <DelegationTokenIdentifier> dtSecretManager) {
    this.dtSecretManager = dtSecretManager;
  }

  public AbstractDelegationTokenSecretManager<DelegationTokenIdentifier>
      getSecretManager() {
    return this.dtSecretManager;
  }

  public void stop() {
    LOG.info("Stopping security manager");
    if(this.dtSecretManager != null) {

View on GitHub (pinned to 2add963021)