{"record":{"id":"bce1e36b313f1a86","repo":"apache/hadoop","slug":"could-not-set-shared-counter","errorCode":null,"errorMessage":"Could not set shared counter !!","messagePattern":"Could not set shared counter !!","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java","lineNumber":532,"sourceCode":"      } catch (InterruptedException e) {\n        // The ExpirationThread is just finishing.. so dont do anything..\n        LOG.debug(\n            \"Thread interrupted while performing token counter increment\", e);\n        Thread.currentThread().interrupt();\n      } catch (Exception e) {\n        throw new RuntimeException(\"Could not increment shared counter !!\", e);\n      }\n    }\n\n    return ++currentSeqNum;\n  }\n\n  @Override\n  protected void setDelegationTokenSeqNum(int seqNum) {\n    try {\n      delTokSeqCounter.setCount(seqNum);\n    } catch (Exception e) {\n      throw new RuntimeException(\"Could not set shared counter !!\", e);\n    }\n  }\n\n  @Override\n  protected int getCurrentKeyId() {\n    return keyIdSeqCounter.getCount();\n  }\n\n  @Override\n  protected int incrementCurrentKeyId() {\n    try {\n      incrSharedCount(keyIdSeqCounter, 1);\n    } catch (InterruptedException e) {\n      // The ExpirationThread is just finishing.. so dont do anything..\n      LOG.debug(\"Thread interrupted while performing keyId increment\", e);\n      Thread.currentThread().interrupt();\n    } catch (Exception e) {\n      throw new RuntimeException(\"Could not increment shared keyId counter !!\", e);","sourceCodeStart":514,"sourceCodeEnd":550,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java#L514-L550","documentation":"setDelegationTokenSeqNum(int) pushes a value into the ZooKeeper SharedCount backing /ZKDTSMSeqNumRoot (delTokSeqCounter.setCount). It is invoked by AbstractDelegationTokenSecretManager.reset() (writes 0 when all state is cleared) and when reloading persisted token identifiers (setDelegationTokenSeqNum(identifier.getSequenceNumber()), AbstractDelegationTokenSecretManager.java:476) so the counter never regresses after restart. Any ZK error during setCount is wrapped as RuntimeException \"Could not set shared counter !!\".","triggerScenarios":"Calling reset() on the secret manager (test harnesses, failover re-init) or restoring token state from the store while ZK is unreachable, the session has expired, or the counter znode's ACL denies writes to the current principal.","commonSituations":"Restart/recovery of an HA service racing a ZK outage or rolling maintenance; principal change between deployments leaving /ZKDTSMSeqNumRoot with stale ACLs; someone deleting the counter znode between stop and start.","solutions":["Confirm ZK is up and the service can write /zkdtsm/ZKDTSMRoot/ZKDTSMSeqNumRoot before restarting the service.","Retry the operation that triggered the load/reset once the quorum is healthy - setCount is idempotent at the value level.","Never delete ZKDTSMSeqNumRoot or other ZKDTSM znodes while any peer service is running; stop all owners first.","Re-check kerberos.keytab/principal and JAAS config if NoAuth is the root cause in the exception chain."],"exampleFix":"// before\nprotected void setDelegationTokenSeqNum(int seqNum) {\n  try { delTokSeqCounter.setCount(seqNum); }\n  catch (Exception e) { throw new RuntimeException(\"Could not set shared counter !!\", e); }\n}\n// after (defensive caller): only reset/restore once the ZK session is usable\nif (zkClient.getZookeeperClient().isConnected()) {\n  secretManager.setDelegationTokenSeqNum(seqNum);\n} else {\n  throw new IOException(\"ZK not connected; cannot restore seq num \" + seqNum);\n}","handlingStrategy":"retry","validationCode":"// Guard reset()/restore paths: only touch the shared counter with a live session\nPreconditions.checkState(zkClient.getZookeeperClient().isConnected(),\n    \"ZK session down; cannot set delegation token seq num\");","typeGuard":null,"tryCatchPattern":"try {\n  secretManager.setDelegationTokenSeqNum(seq);\n} catch (RuntimeException e) {\n  Throwable k = e.getCause();\n  if (k instanceof KeeperException && ((KeeperException) k).code().intValue() != 0) {\n    // wait for quorum, then re-run the restore from the persisted identifiers\n  } else { throw e; }\n}","preventionTips":["Perform secret-manager resets/restores only while a ZK session is verified healthy.","Sequence stop-all -> reset ZK state -> start-all when re-basing token state.","Watch logs for the 'Could not set shared counter' marker during recovery drills."],"tags":["zookeeper","delegation-token","shared-count","recovery"],"backgroundTag":"zookeeper-counter-update-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}