apache/hadoop · error · RuntimeException

Could not set shared counter !!

Error message

Could not set shared counter !!

What it means

setDelegationTokenSeqNum(int) pushes a value into the ZooKeeper SharedCount backing /ZKDTSMSeqNumRoot (delTokSeqCounter.setCount). It is invoked by AbstractDelegationTokenSecretManager.reset() (writes 0 when all state is cleared) and when reloading persisted token identifiers (setDelegationTokenSeqNum(identifier.getSequenceNumber()), AbstractDelegationTokenSecretManager.java:476) so the counter never regresses after restart. Any ZK error during setCount is wrapped as RuntimeException "Could not set shared counter !!".

Solutions

1. Confirm ZK is up and the service can write /zkdtsm/ZKDTSMRoot/ZKDTSMSeqNumRoot before restarting the service.
2. Retry the operation that triggered the load/reset once the quorum is healthy - setCount is idempotent at the value level.
3. Never delete ZKDTSMSeqNumRoot or other ZKDTSM znodes while any peer service is running; stop all owners first.
4. Re-check kerberos.keytab/principal and JAAS config if NoAuth is the root cause in the exception chain.

Example fix

// before
protected void setDelegationTokenSeqNum(int seqNum) {
  try { delTokSeqCounter.setCount(seqNum); }
  catch (Exception e) { throw new RuntimeException("Could not set shared counter !!", e); }
}
// after (defensive caller): only reset/restore once the ZK session is usable
if (zkClient.getZookeeperClient().isConnected()) {
  secretManager.setDelegationTokenSeqNum(seqNum);
} else {
  throw new IOException("ZK not connected; cannot restore seq num " + seqNum);
}

// Guard reset()/restore paths: only touch the shared counter with a live session
Preconditions.checkState(zkClient.getZookeeperClient().isConnected(),
    "ZK session down; cannot set delegation token seq num");

try {
  secretManager.setDelegationTokenSeqNum(seq);
} catch (RuntimeException e) {
  Throwable k = e.getCause();
  if (k instanceof KeeperException && ((KeeperException) k).code().intValue() != 0) {
    // wait for quorum, then re-run the restore from the persisted identifiers
  } else { throw e; }
}

When it happens

Trigger: Calling reset() on the secret manager (test harnesses, failover re-init) or restoring token state from the store while ZK is unreachable, the session has expired, or the counter znode's ACL denies writes to the current principal.

Common situations: Restart/recovery of an HA service racing a ZK outage or rolling maintenance; principal change between deployments leaving /ZKDTSMSeqNumRoot with stale ACLs; someone deleting the counter znode between stop and start.

Related errors

• Could not increment shared counter !!
• Could not increment shared keyId counter !!
• Could not remove Stored Token ZKDTSMDelegationToken_
• Could not start
• Could not update Stored Token ZKDTSMDelegationToken_

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/bce1e36b313f1a86. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java:532

      } catch (InterruptedException e) {
        // The ExpirationThread is just finishing.. so dont do anything..
        LOG.debug(
            "Thread interrupted while performing token counter increment", e);
        Thread.currentThread().interrupt();
      } catch (Exception e) {
        throw new RuntimeException("Could not increment shared counter !!", e);
      }
    }

    return ++currentSeqNum;
  }

  @Override
  protected void setDelegationTokenSeqNum(int seqNum) {
    try {
      delTokSeqCounter.setCount(seqNum);
    } catch (Exception e) {
      throw new RuntimeException("Could not set shared counter !!", e);
    }
  }

  @Override
  protected int getCurrentKeyId() {
    return keyIdSeqCounter.getCount();
  }

  @Override
  protected int incrementCurrentKeyId() {
    try {
      incrSharedCount(keyIdSeqCounter, 1);
    } catch (InterruptedException e) {
      // The ExpirationThread is just finishing.. so dont do anything..
      LOG.debug("Thread interrupted while performing keyId increment", e);
      Thread.currentThread().interrupt();
    } catch (Exception e) {
      throw new RuntimeException("Could not increment shared keyId counter !!", e);

View on GitHub (pinned to 2add963021)