apache/hadoop · critical · IOException

Failed to create SecretManager

Error message

Failed to create SecretManager

What it means

RouterSecurityManager is created when hadoop.security.authentication=kerberos; it instantiates the Router's DelegationTokenSecretManager via FederationUtil.newSecretManager using the class in dfs.federation.router.delegation-token.driver-class. If the returned instance is null (class could not be instantiated) or isRunning() is false (its threads were never started / startThreads failed), the manager cannot issue tokens for the federation, so the constructor throws this IOException and the Router fails to start.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/security/RouterSecurityManager.java:62

 * Manager to hold underlying delegation token secret manager implementations.
 */
public class RouterSecurityManager {

  private static final Logger LOG =
      LoggerFactory.getLogger(RouterSecurityManager.class);

  private AbstractDelegationTokenSecretManager<DelegationTokenIdentifier>
      dtSecretManager = null;

  public RouterSecurityManager(Configuration conf) throws IOException {
    AuthenticationMethod authMethodConfigured =
        SecurityUtil.getAuthenticationMethod(conf);
    AuthenticationMethod authMethodToInit =
        AuthenticationMethod.KERBEROS;
    if (authMethodConfigured.equals(authMethodToInit)) {
      this.dtSecretManager = FederationUtil.newSecretManager(conf);
      if (this.dtSecretManager == null || !this.dtSecretManager.isRunning()) {
        throw new IOException("Failed to create SecretManager");
      }
    }
  }

  @VisibleForTesting
  public RouterSecurityManager(AbstractDelegationTokenSecretManager
      <DelegationTokenIdentifier> dtSecretManager) {
    this.dtSecretManager = dtSecretManager;
  }

  public AbstractDelegationTokenSecretManager<DelegationTokenIdentifier>
      getSecretManager() {
    return this.dtSecretManager;
  }

  public void stop() {
    LOG.info("Stopping security manager");
    if(this.dtSecretManager != null) {

View on GitHub (pinned to 2add963021)

Solutions

1. Read the log lines immediately before this exception — FederationUtil.newSecretManager and the driver implementation log the underlying instantiation/startThreads failure (SQL connect error, ZK connect error, ClassNotFoundException).
2. Fix the backend the token driver depends on: verify MySQL connectivity/credentials/schema or ZooKeeper reachability from the router host.
3. Verify dfs.federation.router.delegation-token.driver-class names a class on the router classpath that extends AbstractDelegationTokenSecretManager.
4. If you do not need Router-issued delegation tokens, leave the default driver and ensure kerberos configuration is otherwise complete.
5. Restart the Router once the backend is healthy — this check runs only at startup.

// Router startup / tooling that instantiates the security manager:
try {
  RouterSecurityManager sm = new RouterSecurityManager(conf);
} catch (IOException e) {
  if ("Failed to create SecretManager".equals(e.getMessage())) {
    // token driver backend (MySQL/ZK) or driver class is broken: fail startup loudly,
    // check earlier log lines for the underlying instantiation/startThreads error
    throw new IllegalStateException("Router token secret manager unavailable; check "
        + "dfs.federation.router.delegation-token.driver-class backend", e);
  }
  throw e;
}

When it happens

Trigger: Instantiating RouterSecurityManager with kerberos enabled while the configured token driver class is missing, mistyped, not an AbstractDelegationTokenSecretManager, or throws in its constructor; or the driver (SQL/ZooKeeper-backed impl) constructs but startThreads fails because its backend (MySQL, ZooKeeper) is unreachable — its own wrapper exception surfaces right before this one.

Common situations: Switching dfs.federation.router.delegation-token.driver-class to SQLDelegationTokenSecretManagerImpl or ZKDelegationTokenSecretManagerImpl without provisioning the MySQL schema or ZK ensemble; wrong JDBC credentials/URL; ZK connection string wrong; class not on the router classpath after a custom build; MySQL down at router restart with the SQL driver configured.

Related errors

• Delegation Token can be issued only with kerberos or web aut
• Delegation Token can be renewed only with kerberos or web au
• Fetch of delegation token failed
• Security is enabled but block access tokens (via dfs.block.a
• Unable to bind on specified streaming port in secure context

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/20b1ffe2676fc5a9. Report an issue: GitHub.