apache/hadoop · error · RuntimeException

Could not increment shared keyId counter !!

Error message

Could not increment shared keyId counter !!

What it means

incrementCurrentKeyId() advances the master-key id held in a Curator SharedCount on /ZKDTSMKeyIdRoot (batch of 1). It runs whenever AbstractDelegationTokenSecretManager rolls a new delegation key (updateCurrentKey on keyUpdateInterval, or a missing key lookup). ZK failure is wrapped as RuntimeException "Could not increment shared keyId counter !!"; the InterruptedException branch is benign (thread shutdown only). Because rollNewKey runs on the key-updater thread, this exception kills that thread and stalls new token issuance.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java:550

      throw new RuntimeException("Could not set shared counter !!", e);
    }
  }

  @Override
  protected int getCurrentKeyId() {
    return keyIdSeqCounter.getCount();
  }

  @Override
  protected int incrementCurrentKeyId() {
    try {
      incrSharedCount(keyIdSeqCounter, 1);
    } catch (InterruptedException e) {
      // The ExpirationThread is just finishing.. so dont do anything..
      LOG.debug("Thread interrupted while performing keyId increment", e);
      Thread.currentThread().interrupt();
    } catch (Exception e) {
      throw new RuntimeException("Could not increment shared keyId counter !!", e);
    }
    return keyIdSeqCounter.getCount();
  }

  @Override
  protected DelegationKey getDelegationKey(int keyId) {
    // First check if its I already have this key
    DelegationKey key = allKeys.get(keyId);
    // Then query ZK
    if (key == null) {
      try {
        key = getKeyFromZK(keyId);
        if (key != null) {
          allKeys.put(keyId, key);
        }
      } catch (IOException e) {
        LOG.error("Error retrieving key [" + keyId + "] from ZK", e);
      }

View on GitHub (pinned to 2add963021)

Solutions

1. Restore ZK connectivity, then restart the affected service so the key-updater thread resumes (it does not self-heal once dead).
2. Verify the service principal can write /zkdtsm/ZKDTSMRoot/ZKDTSMKeyIdRoot with zkCli.
3. Align zk-dt-secret-manager.kerberos.* / JAAS settings across all HA peers.
4. Increase zkNumRetries/session timeouts to survive transient quorum loss.

Example fix

// before: key rolls fail hard during a ZK blip and the updater thread dies
// after (caller-side resilience): monitor and restart the manager when ZK recovers
try {
  secretManager.startThreads(); // re-arms keyId SharedCount and updater thread
} catch (IOException e) {
  LOG.warn("Failed to restart token secret manager after ZK recovery", e);
}

// Before a manual key roll, verify the keyId counter path exists and is writable
Stat s = zkClient.checkExists().forPath("/zkdtsm/ZKDTSMRoot/ZKDTSMKeyIdRoot");
if (s == null) throw new IOException("ZKDTSMKeyIdRoot missing; restart manager to recreate");

try {
  int keyId = secretManager.getCurrentKeyId();
} catch (RuntimeException e) { // e.g. from key roll on dead ZK thread
  // after ZK recovers: restart the owning service to re-arm the key-updater thread
}

When it happens

Trigger: The key-roll timer fires while ZK is unreachable; SharedCount.trySetCount on /ZKDTSMKeyIdRoot throws ConnectionLoss/SessionExpired/NoAuth; the keyId znode was deleted or its ACL changed under the service principal.

Common situations: ZK maintenance window overlapping keyUpdateInterval; expired Kerberos credentials for the ZK client after keytab rotation; HA peers configured with different JAAS principals; long GC pause causing session timeout.

Related errors

• Could not increment shared counter !!
• Unexpected ZooKeeper issue fetching active node info
• ${nodePath} znode could not be created !!
• Could not set shared counter !!
• Could not update Stored Token ZKDTSMDelegationToken_${sequen

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/8340fb6cb40a0926. Report an issue: GitHub.