{"record":{"id":"8340fb6cb40a0926","repo":"apache/hadoop","slug":"could-not-increment-shared-keyid-counter","errorCode":null,"errorMessage":"Could not increment shared keyId counter !!","messagePattern":"Could not increment shared keyId counter !!","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java","lineNumber":550,"sourceCode":"      throw new RuntimeException(\"Could not set shared counter !!\", e);\n    }\n  }\n\n  @Override\n  protected int getCurrentKeyId() {\n    return keyIdSeqCounter.getCount();\n  }\n\n  @Override\n  protected int incrementCurrentKeyId() {\n    try {\n      incrSharedCount(keyIdSeqCounter, 1);\n    } catch (InterruptedException e) {\n      // The ExpirationThread is just finishing.. so dont do anything..\n      LOG.debug(\"Thread interrupted while performing keyId increment\", e);\n      Thread.currentThread().interrupt();\n    } catch (Exception e) {\n      throw new RuntimeException(\"Could not increment shared keyId counter !!\", e);\n    }\n    return keyIdSeqCounter.getCount();\n  }\n\n  @Override\n  protected DelegationKey getDelegationKey(int keyId) {\n    // First check if its I already have this key\n    DelegationKey key = allKeys.get(keyId);\n    // Then query ZK\n    if (key == null) {\n      try {\n        key = getKeyFromZK(keyId);\n        if (key != null) {\n          allKeys.put(keyId, key);\n        }\n      } catch (IOException e) {\n        LOG.error(\"Error retrieving key [\" + keyId + \"] from ZK\", e);\n      }","sourceCodeStart":532,"sourceCodeEnd":568,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java#L532-L568","documentation":"incrementCurrentKeyId() advances the master-key id held in a Curator SharedCount on /ZKDTSMKeyIdRoot (batch of 1). It runs whenever AbstractDelegationTokenSecretManager rolls a new delegation key (updateCurrentKey on keyUpdateInterval, or a missing key lookup). ZK failure is wrapped as RuntimeException \"Could not increment shared keyId counter !!\"; the InterruptedException branch is benign (thread shutdown only). Because rollNewKey runs on the key-updater thread, this exception kills that thread and stalls new token issuance.","triggerScenarios":"The key-roll timer fires while ZK is unreachable; SharedCount.trySetCount on /ZKDTSMKeyIdRoot throws ConnectionLoss/SessionExpired/NoAuth; the keyId znode was deleted or its ACL changed under the service principal.","commonSituations":"ZK maintenance window overlapping keyUpdateInterval; expired Kerberos credentials for the ZK client after keytab rotation; HA peers configured with different JAAS principals; long GC pause causing session timeout.","solutions":["Restore ZK connectivity, then restart the affected service so the key-updater thread resumes (it does not self-heal once dead).","Verify the service principal can write /zkdtsm/ZKDTSMRoot/ZKDTSMKeyIdRoot with zkCli.","Align zk-dt-secret-manager.kerberos.* / JAAS settings across all HA peers.","Increase zkNumRetries/session timeouts to survive transient quorum loss."],"exampleFix":"// before: key rolls fail hard during a ZK blip and the updater thread dies\n// after (caller-side resilience): monitor and restart the manager when ZK recovers\ntry {\n  secretManager.startThreads(); // re-arms keyId SharedCount and updater thread\n} catch (IOException e) {\n  LOG.warn(\"Failed to restart token secret manager after ZK recovery\", e);\n}","handlingStrategy":"retry","validationCode":"// Before a manual key roll, verify the keyId counter path exists and is writable\nStat s = zkClient.checkExists().forPath(\"/zkdtsm/ZKDTSMRoot/ZKDTSMKeyIdRoot\");\nif (s == null) throw new IOException(\"ZKDTSMKeyIdRoot missing; restart manager to recreate\");","typeGuard":null,"tryCatchPattern":"try {\n  int keyId = secretManager.getCurrentKeyId();\n} catch (RuntimeException e) { // e.g. from key roll on dead ZK thread\n  // after ZK recovers: restart the owning service to re-arm the key-updater thread\n}","preventionTips":["Keep ZK session timeouts comfortably larger than host GC pause ceilings.","Alert on the death of key-updater/expiration threads (new token issuance failure is the downstream symptom).","Rotate keytabs for the ZK client principal atomically across HA peers."],"tags":["zookeeper","delegation-key","shared-count","key-roll","ha"],"backgroundTag":"zookeeper-counter-update-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}