apache/hadoop · error · IllegalStateException

Counter table not initialized: {table}

Error message

Counter table not initialized: {table}

What it means

DistributedSQLCounter expects a pre-provisioned single-row table ('One record must exist on the table at all times') — LastSequenceNum(sequenceNum) and LastDelegationKeyId(keyId) for the SQL-backed Router token secret manager. selectCounterValue runs 'SELECT <field> FROM <table> [FOR UPDATE]'; if the ResultSet has no rows it throws IllegalStateException('Counter table not initialized: <table>'). It fires while reserving token sequence numbers or delegation key ids, typically at Router startup or first token operation, and callers like SQLDelegationTokenSecretManager wrap it into RuntimeException.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/security/token/DistributedSQLCounter.java:72

   * @return counter value.
   * @throws SQLException if querying the database fails.
   */
  public int selectCounterValue() throws SQLException {
    try (Connection connection = connectionFactory.getConnection()) {
      return selectCounterValue(false, connection);
    }
  }

  private int selectCounterValue(boolean forUpdate, Connection connection) throws SQLException {
    String query = String.format("SELECT %s FROM %s %s", field, table,
        forUpdate ? "FOR UPDATE" : "");
    LOG.debug("Select counter statement: " + query);
    try (Statement statement = connection.createStatement();
        ResultSet result = statement.executeQuery(query)) {
      if (result.next()) {
        return result.getInt(field);
      } else {
        throw new IllegalStateException("Counter table not initialized: " + table);
      }
    }
  }

  /**
   * Sets the counter to the given value.
   *
   * @param value Value to assign to counter.
   * @throws SQLException if querying the database fails.
   */
  public void updateCounterValue(int value) throws SQLException {
    try (Connection connection = connectionFactory.getConnection(true)) {
      updateCounterValue(value, connection);
    }
  }

  /**
   * Sets the counter to the given value.

View on GitHub (pinned to 2add963021)

Solutions

1. Seed the counter rows: INSERT INTO LastSequenceNum(sequenceNum) VALUES (0); INSERT INTO LastDelegationKeyId(keyId) VALUES (0); — if tokens/keys already exist, seed with SELECT MAX(sequenceNum) FROM Tokens / MAX(keyId) FROM DelegationKeys to avoid duplicate ids.
2. Confirm the tables the JDBC connection actually targets: run SHOW TABLES and SELECT * against the same URL/user the Router uses.
3. Restart the Router (or retry the token operation) after seeding — the counters are read at startup and on batch exhaustion.
4. Guard the provisioning script so DDL and seed INSERT always ship together.

Example fix

-- before: table created but empty
CREATE TABLE LastSequenceNum (sequenceNum INT NOT NULL);

-- after: create and seed the mandatory single row
CREATE TABLE LastSequenceNum (sequenceNum INT NOT NULL);
INSERT INTO LastSequenceNum (sequenceNum) VALUES (0);

// Pre-flight: counter tables must contain exactly one seeded row
try (Connection c = dataSource.getConnection();
     Statement s = c.createStatement();
     ResultSet rs = s.executeQuery("SELECT COUNT(*) FROM LastSequenceNum")) {
  rs.next();
  if (rs.getInt(1) != 1) {
    throw new IllegalStateException("LastSequenceNum not seeded: run the seed INSERT");
  }
}

try {
  int seq = secretManager.getDelegationTokenSeqNum();
} catch (RuntimeException e) {
  if (e.getCause() instanceof IllegalStateException
      && e.getCause().getMessage().contains("Counter table not initialized")) {
    // seed LastSequenceNum/LastDelegationKeyId and retry, don't fabricate ids
    throw new provisioning error pointing at the seed script;
  }
  throw e;
}

When it happens

Trigger: The counter tables exist but were never seeded with their initial row (DDL ran, seed INSERT skipped); someone truncated or deleted the counter rows; the JDBC URL points at a fresh/wrong schema so the SELECT hits an empty table; table-name case mismatch (e.g. Linux MySQL lower_case_table_names) resolving to a different empty table.

Common situations: First rollout of the MySQL-backed Router delegation token store where the schema script was applied without its seed statements; DBA cleanup that emptied 'small' tables; environment migration to a new database that recreated structure but not data.

Related errors

• Error starting threads for MySQL secret manager
• Cannot fetch records for {clazz}
• Failed to create SecretManager
• Delegation Token can be issued only with kerberos or web aut
• Delegation Token can be renewed only with kerberos or web au

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ac829cd76f6c33f3. Report an issue: GitHub.