apache/hadoop · error · RuntimeException

Could not set shared counter !!

Error message

Could not set shared counter !!

What it means

setDelegationTokenSeqNum(int) pushes a value into the ZooKeeper SharedCount backing /ZKDTSMSeqNumRoot (delTokSeqCounter.setCount). It is invoked by AbstractDelegationTokenSecretManager.reset() (writes 0 when all state is cleared) and when reloading persisted token identifiers (setDelegationTokenSeqNum(identifier.getSequenceNumber()), AbstractDelegationTokenSecretManager.java:476) so the counter never regresses after restart. Any ZK error during setCount is wrapped as RuntimeException "Could not set shared counter !!".

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java:532

      } catch (InterruptedException e) {
        // The ExpirationThread is just finishing.. so dont do anything..
        LOG.debug(
            "Thread interrupted while performing token counter increment", e);
        Thread.currentThread().interrupt();
      } catch (Exception e) {
        throw new RuntimeException("Could not increment shared counter !!", e);
      }
    }

    return ++currentSeqNum;
  }

  @Override
  protected void setDelegationTokenSeqNum(int seqNum) {
    try {
      delTokSeqCounter.setCount(seqNum);
    } catch (Exception e) {
      throw new RuntimeException("Could not set shared counter !!", e);
    }
  }

  @Override
  protected int getCurrentKeyId() {
    return keyIdSeqCounter.getCount();
  }

  @Override
  protected int incrementCurrentKeyId() {
    try {
      incrSharedCount(keyIdSeqCounter, 1);
    } catch (InterruptedException e) {
      // The ExpirationThread is just finishing.. so dont do anything..
      LOG.debug("Thread interrupted while performing keyId increment", e);
      Thread.currentThread().interrupt();
    } catch (Exception e) {
      throw new RuntimeException("Could not increment shared keyId counter !!", e);

View on GitHub (pinned to 2add963021)

Solutions

1. Confirm ZK is up and the service can write /zkdtsm/ZKDTSMRoot/ZKDTSMSeqNumRoot before restarting the service.
2. Retry the operation that triggered the load/reset once the quorum is healthy - setCount is idempotent at the value level.
3. Never delete ZKDTSMSeqNumRoot or other ZKDTSM znodes while any peer service is running; stop all owners first.
4. Re-check kerberos.keytab/principal and JAAS config if NoAuth is the root cause in the exception chain.

Example fix

// before
protected void setDelegationTokenSeqNum(int seqNum) {
  try { delTokSeqCounter.setCount(seqNum); }
  catch (Exception e) { throw new RuntimeException("Could not set shared counter !!", e); }
}
// after (defensive caller): only reset/restore once the ZK session is usable
if (zkClient.getZookeeperClient().isConnected()) {
  secretManager.setDelegationTokenSeqNum(seqNum);
} else {
  throw new IOException("ZK not connected; cannot restore seq num " + seqNum);
}

// Guard reset()/restore paths: only touch the shared counter with a live session
Preconditions.checkState(zkClient.getZookeeperClient().isConnected(),
    "ZK session down; cannot set delegation token seq num");

try {
  secretManager.setDelegationTokenSeqNum(seq);
} catch (RuntimeException e) {
  Throwable k = e.getCause();
  if (k instanceof KeeperException && ((KeeperException) k).code().intValue() != 0) {
    // wait for quorum, then re-run the restore from the persisted identifiers
  } else { throw e; }
}

When it happens

Trigger: Calling reset() on the secret manager (test harnesses, failover re-init) or restoring token state from the store while ZK is unreachable, the session has expired, or the counter znode's ACL denies writes to the current principal.

Common situations: Restart/recovery of an HA service racing a ZK outage or rolling maintenance; principal change between deployments leaving /ZKDTSMSeqNumRoot with stale ACLs; someone deleting the counter znode between stop and start.

Related errors

• Could not increment shared counter !!
• ${nodePath} znode could not be created !!
• Could not increment shared keyId counter !!
• Could not update Stored Token ZKDTSMDelegationToken_${sequen
• Could not remove Stored Token ZKDTSMDelegationToken_${sequen

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/bce1e36b313f1a86. Report an issue: GitHub.