{"record":{"id":"92965948a0cd0e09","repo":"apache/hadoop","slug":"could-not-increment-shared-counter","errorCode":null,"errorMessage":"Could not increment shared counter !!","messagePattern":"Could not increment shared counter !!","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java","lineNumber":520,"sourceCode":"  @Override\n  protected int incrementDelegationTokenSeqNum() {\n    // The secret manager will keep a local range of seq num which won't be\n    // seen by peers, so only when the range is exhausted it will ask zk for\n    // another range again\n    if (currentSeqNum >= currentMaxSeqNum) {\n      try {\n        // after a successful batch request, we can get the range starting point\n        currentSeqNum = incrSharedCount(delTokSeqCounter, seqNumBatchSize);\n        currentMaxSeqNum = currentSeqNum + seqNumBatchSize;\n        LOG.info(\"Fetched new range of seq num, from {} to {} \",\n            currentSeqNum+1, currentMaxSeqNum);\n      } catch (InterruptedException e) {\n        // The ExpirationThread is just finishing.. so dont do anything..\n        LOG.debug(\n            \"Thread interrupted while performing token counter increment\", e);\n        Thread.currentThread().interrupt();\n      } catch (Exception e) {\n        throw new RuntimeException(\"Could not increment shared counter !!\", e);\n      }\n    }\n\n    return ++currentSeqNum;\n  }\n\n  @Override\n  protected void setDelegationTokenSeqNum(int seqNum) {\n    try {\n      delTokSeqCounter.setCount(seqNum);\n    } catch (Exception e) {\n      throw new RuntimeException(\"Could not set shared counter !!\", e);\n    }\n  }\n\n  @Override\n  protected int getCurrentKeyId() {\n    return keyIdSeqCounter.getCount();","sourceCodeStart":502,"sourceCodeEnd":538,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java#L502-L538","documentation":"The ZK-backed token manager allocates sequence numbers in batches: a Curator SharedCount on /ZKDTSMSeqNumRoot is incremented by seqNumBatchSize whenever the local range is exhausted. incrSharedCount() only loops on version conflicts (contention with HA peers); any other ZK error escapes and is wrapped as RuntimeException \"Could not increment shared counter !!\". InterruptedException is swallowed separately because it only fires while the expiration thread is shutting down.","triggerScenarios":"Issuing delegation tokens until currentSeqNum reaches currentMaxSeqNum, then incrSharedCount(delTokSeqCounter, seqNumBatchSize) hits ConnectionLoss, SessionExpiredException, SystemErrorException or NoAuth from SharedCount.trySetCount; also the initial batch allocation during startThreads() (line 269) failing the same way.","commonSituations":"ZK session expiry during a token-issuance burst; ZK quorum partition while HA RMs/NNs stay up; auth regression after a keytab rollover; /ZKDTSMSeqNumRoot znode deleted manually while services run; zk-dt-secret-manager.zkNumRetries too small for slow links.","solutions":["Restore ZK connectivity and retry the token operation - the counter value in ZK stays consistent because only successful trySetCount commits.","Verify the service principal still has write ACL on /zkdtsm/ZKDTSMRoot/ZKDTSMSeqNumRoot after keytab/principal changes.","Raise zk-dt-secret-manager.zkNumRetries / zkSessionTimeout so Curator rides out transient connection loss.","If the counter znode was deleted, restart the owning service so startThreads() recreates it, then retry."],"exampleFix":"// before: default retries give up on a brief partition\n// zk-dt-secret-manager.zkNumRetries = 3 (default)\n<property>\n  <name>zk-dt-secret-manager.zkNumRetries</name>\n  <value>10</value>\n</property>\n<property>\n  <name>zk-dt-secret-manager.zkSessionTimeout</name>\n  <value>60000</value>\n</property>\n// after: token issuance survives short ZK interruptions without the\n// \"Could not increment shared counter !!\" RuntimeException","handlingStrategy":"retry","validationCode":"// Before issuing tokens in bulk, confirm the seqnum counter is reachable\nif (!zkClient.getZookeeperClient().isConnected()) {\n  throw new IOException(\"ZK not connected; defer token issuance\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  Token<?> t = tokenManager.createToken(ugi, renewer, service);\n} catch (RuntimeException e) {\n  if (e.getMessage().contains(\"Could not increment shared counter\")) {\n    // transient ZK failure: back off and retry the issuance; ZK state stays consistent\n  } else { throw e; }\n}","preventionTips":["Size zk-dt-secret-manager.zkNumRetries/zkSessionTimeout for your worst-case ZK recovery window.","Monitor ZK session expiry counts on hosts running token managers.","Never hand-delete /ZKDTSMSeqNumRoot while any HA peer is live."],"tags":["zookeeper","delegation-token","shared-count","token-issue","ha"],"backgroundTag":"zookeeper-counter-update-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}