{"record":{"id":"f056313b22eeddcf","repo":"apache/hadoop","slug":"error-starting-threads-for-mysql-secret-manager","errorCode":null,"errorMessage":"Error starting threads for MySQL secret manager","messagePattern":"Error starting threads for MySQL secret manager","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"critical","filePath":"hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/security/token/SQLDelegationTokenSecretManagerImpl.java","lineNumber":78,"sourceCode":"    this(conf, new HikariDataSourceConnectionFactory(conf),\n        SQLSecretManagerRetriableHandlerImpl.getInstance(conf));\n  }\n\n  public SQLDelegationTokenSecretManagerImpl(Configuration conf,\n      SQLConnectionFactory connectionFactory, SQLSecretManagerRetriableHandler retryHandler) {\n    super(conf);\n\n    this.connectionFactory = connectionFactory;\n    this.sequenceNumCounter = new DistributedSQLCounter(SEQ_NUM_COUNTER_FIELD,\n        SEQ_NUM_COUNTER_TABLE, connectionFactory);\n    this.delegationKeyIdCounter = new DistributedSQLCounter(KEY_ID_COUNTER_FIELD,\n        KEY_ID_COUNTER_TABLE, connectionFactory);\n    this.retryHandler = retryHandler;\n\n    try {\n      super.startThreads();\n    } catch (IOException e) {\n      throw new RuntimeException(\"Error starting threads for MySQL secret manager\", e);\n    }\n\n    LOG.info(\"MySQL delegation token secret manager instantiated\");\n  }\n\n  @Override\n  public DelegationTokenIdentifier createIdentifier() {\n    return new DelegationTokenIdentifier();\n  }\n\n  @Override\n  public void stopThreads() {\n    super.stopThreads();\n    connectionFactory.shutdown();\n  }\n\n  @Override\n  protected void insertToken(int sequenceNum, byte[] tokenIdentifier, byte[] tokenInfo)","sourceCodeStart":60,"sourceCodeEnd":96,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/security/token/SQLDelegationTokenSecretManagerImpl.java#L60-L96","documentation":"SQLDelegationTokenSecretManagerImpl's constructor calls super.startThreads() eagerly; any IOException from AbstractDelegationTokenSecretManager startup — which for this manager includes initializing key/sequence state through the SQL layer (selectSequenceNum/selectKeyId failures surface as RuntimeException wrapping SQLException) — is rethrown as RuntimeException('Error starting threads for MySQL secret manager'). Because the manager is built inside FederationUtil.newSecretManager during RouterSecurityManager creation (kerberos mode), this aborts Router startup.","triggerScenarios":"MySQL unreachable, credentials wrong, or connection pool (HikariCP) initialization failing at Router start; counter tables LastSequenceNum/LastDelegationKeyId or Tokens/DelegationKeys missing so the startup SELECTs throw SQLException; insufficient DB privileges; SQLState errors during the initial counter reads.","commonSituations":"Router restarted while the database was down or restarting; schema provisioning step skipped in a new environment; JDBC URL/credentials changed but router config not updated; DBA revoked SELECT/UPDATE on the token tables.","solutions":["Inspect the caused-by chain — the original IOException/RuntimeException/SQLException names the exact failing operation (connection refused, table missing, access denied).","Verify MySQL connectivity and credentials from the router host using the exact JDBC configuration the manager uses.","Provision the full schema: Tokens, DelegationKeys, LastSequenceNum, LastDelegationKeyId — including the mandatory seed rows in the counter tables.","Fix the sql-dt-secret-manager.*/JDBC settings in the router configuration, then restart the Router."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  new SQLDelegationTokenSecretManagerImpl(conf); // via driver-class during router init\n} catch (RuntimeException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"Error starting threads\")) {\n    Throwable root = e.getCause(); // original IOException/SQLException = real DB problem\n    throw new IllegalStateException(\"MySQL token store failed to start: \" + root, e);\n  }\n  throw e;\n}","preventionTips":["Gate router startup on a MySQL readiness probe (connect + SELECT 1) using the production JDBC config.","Apply the token schema (Tokens, DelegationKeys, counter tables with seed rows) as a versioned migration before the first router start.","Monitor DB availability; stop/restart routers only after the backend passes the probe."],"tags":["hdfs","router-based-federation","delegation-token","mysql","sql","startup"],"backgroundTag":"database-unavailable","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}