apache/hadoop · critical · RuntimeException

Error starting threads for MySQL secret manager

Error message

Error starting threads for MySQL secret manager

What it means

SQLDelegationTokenSecretManagerImpl's constructor calls super.startThreads() eagerly; any IOException from AbstractDelegationTokenSecretManager startup — which for this manager includes initializing key/sequence state through the SQL layer (selectSequenceNum/selectKeyId failures surface as RuntimeException wrapping SQLException) — is rethrown as RuntimeException('Error starting threads for MySQL secret manager'). Because the manager is built inside FederationUtil.newSecretManager during RouterSecurityManager creation (kerberos mode), this aborts Router startup.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/security/token/SQLDelegationTokenSecretManagerImpl.java:78

    this(conf, new HikariDataSourceConnectionFactory(conf),
        SQLSecretManagerRetriableHandlerImpl.getInstance(conf));
  }

  public SQLDelegationTokenSecretManagerImpl(Configuration conf,
      SQLConnectionFactory connectionFactory, SQLSecretManagerRetriableHandler retryHandler) {
    super(conf);

    this.connectionFactory = connectionFactory;
    this.sequenceNumCounter = new DistributedSQLCounter(SEQ_NUM_COUNTER_FIELD,
        SEQ_NUM_COUNTER_TABLE, connectionFactory);
    this.delegationKeyIdCounter = new DistributedSQLCounter(KEY_ID_COUNTER_FIELD,
        KEY_ID_COUNTER_TABLE, connectionFactory);
    this.retryHandler = retryHandler;

    try {
      super.startThreads();
    } catch (IOException e) {
      throw new RuntimeException("Error starting threads for MySQL secret manager", e);
    }

    LOG.info("MySQL delegation token secret manager instantiated");
  }

  @Override
  public DelegationTokenIdentifier createIdentifier() {
    return new DelegationTokenIdentifier();
  }

  @Override
  public void stopThreads() {
    super.stopThreads();
    connectionFactory.shutdown();
  }

  @Override
  protected void insertToken(int sequenceNum, byte[] tokenIdentifier, byte[] tokenInfo)

View on GitHub (pinned to 2add963021)

Solutions

1. Inspect the caused-by chain — the original IOException/RuntimeException/SQLException names the exact failing operation (connection refused, table missing, access denied).
2. Verify MySQL connectivity and credentials from the router host using the exact JDBC configuration the manager uses.
3. Provision the full schema: Tokens, DelegationKeys, LastSequenceNum, LastDelegationKeyId — including the mandatory seed rows in the counter tables.
4. Fix the sql-dt-secret-manager.*/JDBC settings in the router configuration, then restart the Router.

try {
  new SQLDelegationTokenSecretManagerImpl(conf); // via driver-class during router init
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().contains("Error starting threads")) {
    Throwable root = e.getCause(); // original IOException/SQLException = real DB problem
    throw new IllegalStateException("MySQL token store failed to start: " + root, e);
  }
  throw e;
}

When it happens

Trigger: MySQL unreachable, credentials wrong, or connection pool (HikariCP) initialization failing at Router start; counter tables LastSequenceNum/LastDelegationKeyId or Tokens/DelegationKeys missing so the startup SELECTs throw SQLException; insufficient DB privileges; SQLState errors during the initial counter reads.

Common situations: Router restarted while the database was down or restarting; schema provisioning step skipped in a new environment; JDBC URL/credentials changed but router config not updated; DBA revoked SELECT/UPDATE on the token tables.

Related errors

• Counter table not initialized: {table}
• Failed to create SecretManager
• Cannot fetch records for {clazz}
• Same delegation token being added twice; invalid entry in fs
• Can't update persisted delegation token renewal to a running

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f056313b22eeddcf. Report an issue: GitHub.