{"record":{"id":"ebd6b091860188e8","repo":"apache/hadoop","slug":"could-not-remove-stored-token-zkdtsmdelegationtoke","errorCode":null,"errorMessage":"Could not remove Stored Token ZKDTSMDelegationToken_${sequenceNumber}","messagePattern":"Could not remove Stored Token ZKDTSMDelegationToken_(.+?)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java","lineNumber":821,"sourceCode":"          LOG.debug(\"Removing ZKDTSMDelegationToken_\"\n              + ident.getSequenceNumber());\n        }\n        while(zkClient.checkExists().forPath(nodeRemovePath) != null){\n          try {\n            zkClient.delete().guaranteed().forPath(nodeRemovePath);\n          } catch (NoNodeException nne) {\n            // It is possible that the node might be deleted between the\n            // check and the actual delete.. which might lead to an\n            // exception that can bring down the daemon running this\n            // SecretManager\n            LOG.debug(\"Node already deleted by peer \" + nodeRemovePath);\n          }\n        }\n      } else {\n        LOG.debug(\"Attempted to remove a non-existing znode \" + nodeRemovePath);\n      }\n    } catch (Exception e) {\n      throw new RuntimeException(\n          \"Could not remove Stored Token ZKDTSMDelegationToken_\"\n          + ident.getSequenceNumber(), e);\n    }\n  }\n\n  @Override\n  public TokenIdent cancelToken(Token<TokenIdent> token,\n      String canceller) throws IOException {\n    ByteArrayInputStream buf = new ByteArrayInputStream(token.getIdentifier());\n    DataInputStream in = new DataInputStream(buf);\n    TokenIdent id = createIdentifier();\n    id.readFields(in);\n\n    syncLocalCacheWithZk(id);\n    return super.cancelToken(token, canceller);\n  }\n\n  protected void addOrUpdateToken(TokenIdent ident,","sourceCodeStart":803,"sourceCodeEnd":839,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java#L803-L839","documentation":"removeStoredToken deletes ZKDTSMTokensRoot/ZKDTSMDelegationToken_<seq> when a token expires or is canceled. KeeperException.NoNodeException is deliberately swallowed (a peer HA node may have deleted it first, which must not crash the daemon), but every other failure is wrapped as RuntimeException \"Could not remove Stored Token ZKDTSMDelegationToken_<seq>\". This typically fires on the token-expiration thread, so an unhandled throw terminates that thread.","triggerScenarios":"Token expiry sweep (removeExpiredTokens) or cancelToken hitting ConnectionLoss, SessionExpired, NoAuth or BADACL on the delete(); also checkExists() against a deleted tokens root.","commonSituations":"Extended ZK partition overlapping the expiry scan interval; session expiry from long GC or network blackholes; ACL drift after principal rotation; ZKDTSMTokensRoot removed while services run.","solutions":["Restore ZK connectivity; expired-but-not-removed tokens are re-attempted on the next scan after the manager restarts.","Verify delete permission on the token znode path for the service principal.","If a poisoned znode keeps failing, remove it manually with zkCli while the service is stopped.","Restart the owning service if the expiration thread died, otherwise tokens stop being cleaned and currentTokens grows."],"exampleFix":"// before: any non-NoNode ZK error during expiry kills the expiration thread\n// after (defensive wrapper around removal): treat peer deletion and transient loss separately\ntry {\n  removeStoredToken(ident, true);\n} catch (RuntimeException e) {\n  Throwable c = e.getCause();\n  if (c instanceof KeeperException.NoNodeException) { LOG.debug(\"already removed\"); }\n  else if (c instanceof KeeperException.ConnectionLoss) { LOG.warn(\"will retry next scan\", e); }\n  else { throw e; }\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Mirrors the class's own NoNode tolerance: peers racing deletion are normal.\ntry {\n  secretManager.removeStoredToken(ident, true);\n} catch (RuntimeException e) {\n  Throwable c = e.getCause();\n  if (c instanceof KeeperException.NoNodeException) { /* already gone: success */ }\n  else if (c instanceof KeeperException.ConnectionLoss) { /* retry on next scan */ }\n  else { throw e; }\n}","preventionTips":["Use removeStoredToken(ident, true) so the check-then-delete race with HA peers is handled inside the manager.","Alert if the currentTokens map grows without bound - a dead expiration thread leaks state.","Do not delete ZKDTSMTokensRoot children externally; let expiry do it."],"tags":["zookeeper","delegation-token","token-expiry","token-cancel","ha"],"backgroundTag":"zookeeper-token-delete-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}