apache/hadoop · error · RuntimeException

Could not update Stored Token ZKDTSMDelegationToken_${sequen

Error message

Could not update Stored Token ZKDTSMDelegationToken_${sequenceNumber}

What it means

When a delegation token is renewed, AbstractDelegationTokenSecretManager.renewToken calls updateStoredToken(ident, renewDate) (AbstractDelegationTokenSecretManager.java:442). The ZK impl checks that ZKDTSMTokensRoot/ZKDTSMDelegationToken_<seq> exists, then addOrUpdateToken with overwrite; missing-node falls back to create, but any other failure is wrapped as RuntimeException "Could not update Stored Token ZKDTSMDelegationToken_<seq>" which the HTTP layer surfaces as an error to the renewing client.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java:774

      throw new RuntimeException(e);
    }
  }

  @Override
  protected void updateToken(TokenIdent ident,
      DelegationTokenInformation tokenInfo) throws IOException {
    String nodeRemovePath =
        getNodePath(ZK_DTSM_TOKENS_ROOT, DELEGATION_TOKEN_PREFIX
            + ident.getSequenceNumber());
    try {
      if (zkClient.checkExists().forPath(nodeRemovePath) != null) {
        addOrUpdateToken(ident, tokenInfo, true);
      } else {
        addOrUpdateToken(ident, tokenInfo, false);
        LOG.debug("Attempted to update a non-existing znode " + nodeRemovePath);
      }
    } catch (Exception e) {
      throw new RuntimeException("Could not update Stored Token ZKDTSMDelegationToken_"
          + ident.getSequenceNumber(), e);
    }
  }

  @Override
  protected void removeStoredToken(TokenIdent ident)
      throws IOException {
    removeStoredToken(ident, false);
  }

  protected void removeStoredToken(TokenIdent ident,
      boolean checkAgainstZkBeforeDeletion) throws IOException {
    String nodeRemovePath =
        getNodePath(ZK_DTSM_TOKENS_ROOT, DELEGATION_TOKEN_PREFIX
            + ident.getSequenceNumber());
    try {
      DelegationTokenInformation dtInfo = getTokenInfoFromZK(ident, true);
      if (dtInfo != null) {

View on GitHub (pinned to 2add963021)

Solutions

1. Check ZK health from the service host and retry the renew once connectivity is confirmed (renewal is idempotent from the client's perspective until max lifetime).
2. Inspect the exception cause chain: KeeperException code (ConnectionLoss vs NoAuth vs NoNode) tells you whether it is network or permissions.
3. Fix ACLs on /zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot if NoAuth.
4. If the token znode was deleted, accept that the token is gone: clients must request a new one.

Example fix

// before: single renew attempt fails on a transient ZK error
client.renewDelegationToken(url, authToken, dt, doAs);
// after: retry with backoff, only for transient causes
for (int i = 0; i < 3; i++) {
  try { client.renewDelegationToken(url, authToken, dt, doAs); break; }
  catch (IOException e) {
    if (!(e.getCause() instanceof KeeperException.ConnectionLoss) || i == 2) throw e;
    Thread.sleep(2000L << i);
  }
}

// Client-side: check token validity window before renewing
long now = Time.now();
if (dt != null && now >= getMaxAge(dt)) { /* skip renew; request new token */ }

catch (RuntimeException e) {
  Throwable c = e.getCause();
  if (c instanceof KeeperException.NoNodeException) { /* token gone: re-request */ }
  else if (c instanceof KeeperException.ConnectionLoss) { /* retry renew after backoff */ }
  else { throw e; }
}

When it happens

Trigger: HTTP renewdelegationtoken (or RM/WebHDFS renew API) while the token znode write fails: connection loss after retries, expired ZK session, NoAuth after principal change, or the tokens root deleted underneath the manager.

Common situations: ZK partition during a renewal storm; token renewal right after a failover before the new active's ZK session is stable; stale ACLs on token znodes created by a previous principal.

Related errors

• ${nodePath} znode could not be created !!
• Could not increment shared counter !!
• Could not remove Stored Token ZKDTSMDelegationToken_${sequen
• Unexpected ZooKeeper issue fetching active node info
• Could not set shared counter !!

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/4ad1949ff75bc0fb. Report an issue: GitHub.