apache/hadoop · error · IllegalStateException

Counter table not initialized

Error message

Counter table not initialized: {table}

What it means

DistributedSQLCounter expects a pre-provisioned single-row table ('One record must exist on the table at all times') — LastSequenceNum(sequenceNum) and LastDelegationKeyId(keyId) for the SQL-backed Router token secret manager. selectCounterValue runs 'SELECT <field> FROM <table> [FOR UPDATE]'; if the ResultSet has no rows it throws IllegalStateException('Counter table not initialized: <table>'). It fires while reserving token sequence numbers or delegation key ids, typically at Router startup or first token operation, and callers like SQLDelegationTokenSecretManager wrap it into RuntimeException.

Solutions

  1. Seed the counter rows: INSERT INTO LastSequenceNum(sequenceNum) VALUES (0); INSERT INTO LastDelegationKeyId(keyId) VALUES (0); — if tokens/keys already exist, seed with SELECT MAX(sequenceNum) FROM Tokens / MAX(keyId) FROM DelegationKeys to avoid duplicate ids.
  2. Confirm the tables the JDBC connection actually targets: run SHOW TABLES and SELECT * against the same URL/user the Router uses.
  3. Restart the Router (or retry the token operation) after seeding — the counters are read at startup and on batch exhaustion.
  4. Guard the provisioning script so DDL and seed INSERT always ship together.

Example fix

-- before: table created but empty
CREATE TABLE LastSequenceNum (sequenceNum INT NOT NULL);

-- after: create and seed the mandatory single row
CREATE TABLE LastSequenceNum (sequenceNum INT NOT NULL);
INSERT INTO LastSequenceNum (sequenceNum) VALUES (0);
Defensive patterns

Strategy: validation

Validate before calling

// Pre-flight: counter tables must contain exactly one seeded row
try (Connection c = dataSource.getConnection();
     Statement s = c.createStatement();
     ResultSet rs = s.executeQuery("SELECT COUNT(*) FROM LastSequenceNum")) {
  rs.next();
  if (rs.getInt(1) != 1) {
    throw new IllegalStateException("LastSequenceNum not seeded: run the seed INSERT");
  }
}

Try / catch

try {
  int seq = secretManager.getDelegationTokenSeqNum();
} catch (RuntimeException e) {
  if (e.getCause() instanceof IllegalStateException
      && e.getCause().getMessage().contains("Counter table not initialized")) {
    // seed LastSequenceNum/LastDelegationKeyId and retry, don't fabricate ids
    throw new provisioning error pointing at the seed script;
  }
  throw e;
}

Prevention

When it happens

Trigger: The counter tables exist but were never seeded with their initial row (DDL ran, seed INSERT skipped); someone truncated or deleted the counter rows; the JDBC URL points at a fresh/wrong schema so the SELECT hits an empty table; table-name case mismatch (e.g. Linux MySQL lower_case_table_names) resolving to a different empty table.

Common situations: First rollout of the MySQL-backed Router delegation token store where the schema script was applied without its seed statements; DBA cleanup that emptied 'small' tables; environment migration to a new database that recreated structure but not data.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ac829cd76f6c33f3. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/security/token/DistributedSQLCounter.java:72

   * @return counter value.
   * @throws SQLException if querying the database fails.
   */
  public int selectCounterValue() throws SQLException {
    try (Connection connection = connectionFactory.getConnection()) {
      return selectCounterValue(false, connection);
    }
  }

  private int selectCounterValue(boolean forUpdate, Connection connection) throws SQLException {
    String query = String.format("SELECT %s FROM %s %s", field, table,
        forUpdate ? "FOR UPDATE" : "");
    LOG.debug("Select counter statement: " + query);
    try (Statement statement = connection.createStatement();
        ResultSet result = statement.executeQuery(query)) {
      if (result.next()) {
        return result.getInt(field);
      } else {
        throw new IllegalStateException("Counter table not initialized: " + table);
      }
    }
  }

  /**
   * Sets the counter to the given value.
   *
   * @param value Value to assign to counter.
   * @throws SQLException if querying the database fails.
   */
  public void updateCounterValue(int value) throws SQLException {
    try (Connection connection = connectionFactory.getConnection(true)) {
      updateCounterValue(value, connection);
    }
  }

  /**
   * Sets the counter to the given value.

View on GitHub (pinned to 2add963021)