apache/hadoop · error · RuntimeException

Could not increment shared counter !!

Error message

Could not increment shared counter !!

What it means

The ZK-backed token manager allocates sequence numbers in batches: a Curator SharedCount on /ZKDTSMSeqNumRoot is incremented by seqNumBatchSize whenever the local range is exhausted. incrSharedCount() only loops on version conflicts (contention with HA peers); any other ZK error escapes and is wrapped as RuntimeException "Could not increment shared counter !!". InterruptedException is swallowed separately because it only fires while the expiration thread is shutting down.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java:520

  @Override
  protected int incrementDelegationTokenSeqNum() {
    // The secret manager will keep a local range of seq num which won't be
    // seen by peers, so only when the range is exhausted it will ask zk for
    // another range again
    if (currentSeqNum >= currentMaxSeqNum) {
      try {
        // after a successful batch request, we can get the range starting point
        currentSeqNum = incrSharedCount(delTokSeqCounter, seqNumBatchSize);
        currentMaxSeqNum = currentSeqNum + seqNumBatchSize;
        LOG.info("Fetched new range of seq num, from {} to {} ",
            currentSeqNum+1, currentMaxSeqNum);
      } catch (InterruptedException e) {
        // The ExpirationThread is just finishing.. so dont do anything..
        LOG.debug(
            "Thread interrupted while performing token counter increment", e);
        Thread.currentThread().interrupt();
      } catch (Exception e) {
        throw new RuntimeException("Could not increment shared counter !!", e);
      }
    }

    return ++currentSeqNum;
  }

  @Override
  protected void setDelegationTokenSeqNum(int seqNum) {
    try {
      delTokSeqCounter.setCount(seqNum);
    } catch (Exception e) {
      throw new RuntimeException("Could not set shared counter !!", e);
    }
  }

  @Override
  protected int getCurrentKeyId() {
    return keyIdSeqCounter.getCount();

View on GitHub (pinned to 2add963021)

Solutions

1. Restore ZK connectivity and retry the token operation - the counter value in ZK stays consistent because only successful trySetCount commits.
2. Verify the service principal still has write ACL on /zkdtsm/ZKDTSMRoot/ZKDTSMSeqNumRoot after keytab/principal changes.
3. Raise zk-dt-secret-manager.zkNumRetries / zkSessionTimeout so Curator rides out transient connection loss.
4. If the counter znode was deleted, restart the owning service so startThreads() recreates it, then retry.

Example fix

// before: default retries give up on a brief partition
// zk-dt-secret-manager.zkNumRetries = 3 (default)
<property>
  <name>zk-dt-secret-manager.zkNumRetries</name>
  <value>10</value>
</property>
<property>
  <name>zk-dt-secret-manager.zkSessionTimeout</name>
  <value>60000</value>
</property>
// after: token issuance survives short ZK interruptions without the
// "Could not increment shared counter !!" RuntimeException

// Before issuing tokens in bulk, confirm the seqnum counter is reachable
if (!zkClient.getZookeeperClient().isConnected()) {
  throw new IOException("ZK not connected; defer token issuance");
}

try {
  Token<?> t = tokenManager.createToken(ugi, renewer, service);
} catch (RuntimeException e) {
  if (e.getMessage().contains("Could not increment shared counter")) {
    // transient ZK failure: back off and retry the issuance; ZK state stays consistent
  } else { throw e; }
}

When it happens

Trigger: Issuing delegation tokens until currentSeqNum reaches currentMaxSeqNum, then incrSharedCount(delTokSeqCounter, seqNumBatchSize) hits ConnectionLoss, SessionExpiredException, SystemErrorException or NoAuth from SharedCount.trySetCount; also the initial batch allocation during startThreads() (line 269) failing the same way.

Common situations: ZK session expiry during a token-issuance burst; ZK quorum partition while HA RMs/NNs stay up; auth regression after a keytab rollover; /ZKDTSMSeqNumRoot znode deleted manually while services run; zk-dt-secret-manager.zkNumRetries too small for slow links.

Related errors

• ${nodePath} znode could not be created !!
• Could not set shared counter !!
• Could not increment shared keyId counter !!
• Could not update Stored Token ZKDTSMDelegationToken_${sequen
• Could not remove Stored Token ZKDTSMDelegationToken_${sequen

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/92965948a0cd0e09. Report an issue: GitHub.