{"record":{"id":"4ad1949ff75bc0fb","repo":"apache/hadoop","slug":"could-not-update-stored-token-zkdtsmdelegationtoke","errorCode":null,"errorMessage":"Could not update Stored Token ZKDTSMDelegationToken_${sequenceNumber}","messagePattern":"Could not update Stored Token ZKDTSMDelegationToken_(.+?)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java","lineNumber":774,"sourceCode":"      throw new RuntimeException(e);\n    }\n  }\n\n  @Override\n  protected void updateToken(TokenIdent ident,\n      DelegationTokenInformation tokenInfo) throws IOException {\n    String nodeRemovePath =\n        getNodePath(ZK_DTSM_TOKENS_ROOT, DELEGATION_TOKEN_PREFIX\n            + ident.getSequenceNumber());\n    try {\n      if (zkClient.checkExists().forPath(nodeRemovePath) != null) {\n        addOrUpdateToken(ident, tokenInfo, true);\n      } else {\n        addOrUpdateToken(ident, tokenInfo, false);\n        LOG.debug(\"Attempted to update a non-existing znode \" + nodeRemovePath);\n      }\n    } catch (Exception e) {\n      throw new RuntimeException(\"Could not update Stored Token ZKDTSMDelegationToken_\"\n          + ident.getSequenceNumber(), e);\n    }\n  }\n\n  @Override\n  protected void removeStoredToken(TokenIdent ident)\n      throws IOException {\n    removeStoredToken(ident, false);\n  }\n\n  protected void removeStoredToken(TokenIdent ident,\n      boolean checkAgainstZkBeforeDeletion) throws IOException {\n    String nodeRemovePath =\n        getNodePath(ZK_DTSM_TOKENS_ROOT, DELEGATION_TOKEN_PREFIX\n            + ident.getSequenceNumber());\n    try {\n      DelegationTokenInformation dtInfo = getTokenInfoFromZK(ident, true);\n      if (dtInfo != null) {","sourceCodeStart":756,"sourceCodeEnd":792,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java#L756-L792","documentation":"When a delegation token is renewed, AbstractDelegationTokenSecretManager.renewToken calls updateStoredToken(ident, renewDate) (AbstractDelegationTokenSecretManager.java:442). The ZK impl checks that ZKDTSMTokensRoot/ZKDTSMDelegationToken_<seq> exists, then addOrUpdateToken with overwrite; missing-node falls back to create, but any other failure is wrapped as RuntimeException \"Could not update Stored Token ZKDTSMDelegationToken_<seq>\" which the HTTP layer surfaces as an error to the renewing client.","triggerScenarios":"HTTP renewdelegationtoken (or RM/WebHDFS renew API) while the token znode write fails: connection loss after retries, expired ZK session, NoAuth after principal change, or the tokens root deleted underneath the manager.","commonSituations":"ZK partition during a renewal storm; token renewal right after a failover before the new active's ZK session is stable; stale ACLs on token znodes created by a previous principal.","solutions":["Check ZK health from the service host and retry the renew once connectivity is confirmed (renewal is idempotent from the client's perspective until max lifetime).","Inspect the exception cause chain: KeeperException code (ConnectionLoss vs NoAuth vs NoNode) tells you whether it is network or permissions.","Fix ACLs on /zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot if NoAuth.","If the token znode was deleted, accept that the token is gone: clients must request a new one."],"exampleFix":"// before: single renew attempt fails on a transient ZK error\nclient.renewDelegationToken(url, authToken, dt, doAs);\n// after: retry with backoff, only for transient causes\nfor (int i = 0; i < 3; i++) {\n  try { client.renewDelegationToken(url, authToken, dt, doAs); break; }\n  catch (IOException e) {\n    if (!(e.getCause() instanceof KeeperException.ConnectionLoss) || i == 2) throw e;\n    Thread.sleep(2000L << i);\n  }\n}","handlingStrategy":"retry","validationCode":"// Client-side: check token validity window before renewing\nlong now = Time.now();\nif (dt != null && now >= getMaxAge(dt)) { /* skip renew; request new token */ }","typeGuard":null,"tryCatchPattern":"catch (RuntimeException e) {\n  Throwable c = e.getCause();\n  if (c instanceof KeeperException.NoNodeException) { /* token gone: re-request */ }\n  else if (c instanceof KeeperException.ConnectionLoss) { /* retry renew after backoff */ }\n  else { throw e; }\n}","preventionTips":["Renew at a fraction of renew interval so a transient ZK failure still leaves retry budget.","Clients should fall back to fetching a fresh delegation token when renewal repeatedly fails.","Track KeeperException codes in server logs to distinguish network vs permission causes."],"tags":["zookeeper","delegation-token","token-renew","ha"],"backgroundTag":"zookeeper-token-update-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}