apache/hadoop · error · RuntimeException

Could not remove Stored Token ZKDTSMDelegationToken_${sequen

Error message

Could not remove Stored Token ZKDTSMDelegationToken_${sequenceNumber}

What it means

removeStoredToken deletes ZKDTSMTokensRoot/ZKDTSMDelegationToken_<seq> when a token expires or is canceled. KeeperException.NoNodeException is deliberately swallowed (a peer HA node may have deleted it first, which must not crash the daemon), but every other failure is wrapped as RuntimeException "Could not remove Stored Token ZKDTSMDelegationToken_<seq>". This typically fires on the token-expiration thread, so an unhandled throw terminates that thread.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java:821

          LOG.debug("Removing ZKDTSMDelegationToken_"
              + ident.getSequenceNumber());
        }
        while(zkClient.checkExists().forPath(nodeRemovePath) != null){
          try {
            zkClient.delete().guaranteed().forPath(nodeRemovePath);
          } catch (NoNodeException nne) {
            // It is possible that the node might be deleted between the
            // check and the actual delete.. which might lead to an
            // exception that can bring down the daemon running this
            // SecretManager
            LOG.debug("Node already deleted by peer " + nodeRemovePath);
          }
        }
      } else {
        LOG.debug("Attempted to remove a non-existing znode " + nodeRemovePath);
      }
    } catch (Exception e) {
      throw new RuntimeException(
          "Could not remove Stored Token ZKDTSMDelegationToken_"
          + ident.getSequenceNumber(), e);
    }
  }

  @Override
  public TokenIdent cancelToken(Token<TokenIdent> token,
      String canceller) throws IOException {
    ByteArrayInputStream buf = new ByteArrayInputStream(token.getIdentifier());
    DataInputStream in = new DataInputStream(buf);
    TokenIdent id = createIdentifier();
    id.readFields(in);

    syncLocalCacheWithZk(id);
    return super.cancelToken(token, canceller);
  }

  protected void addOrUpdateToken(TokenIdent ident,

View on GitHub (pinned to 2add963021)

Solutions

Restore ZK connectivity; expired-but-not-removed tokens are re-attempted on the next scan after the manager restarts.
Verify delete permission on the token znode path for the service principal.
If a poisoned znode keeps failing, remove it manually with zkCli while the service is stopped.
Restart the owning service if the expiration thread died, otherwise tokens stop being cleaned and currentTokens grows.

Example fix

// before: any non-NoNode ZK error during expiry kills the expiration thread
// after (defensive wrapper around removal): treat peer deletion and transient loss separately
try {
  removeStoredToken(ident, true);
} catch (RuntimeException e) {
  Throwable c = e.getCause();
  if (c instanceof KeeperException.NoNodeException) { LOG.debug("already removed"); }
  else if (c instanceof KeeperException.ConnectionLoss) { LOG.warn("will retry next scan", e); }
  else { throw e; }
}

Defensive patterns

Strategy: try-catch

Try / catch

// Mirrors the class's own NoNode tolerance: peers racing deletion are normal.
try {
  secretManager.removeStoredToken(ident, true);
} catch (RuntimeException e) {
  Throwable c = e.getCause();
  if (c instanceof KeeperException.NoNodeException) { /* already gone: success */ }
  else if (c instanceof KeeperException.ConnectionLoss) { /* retry on next scan */ }
  else { throw e; }
}

Prevention

Use removeStoredToken(ident, true) so the check-then-delete race with HA peers is handled inside the manager.
Alert if the currentTokens map grows without bound - a dead expiration thread leaks state.
Do not delete ZKDTSMTokensRoot children externally; let expiry do it.

When it happens

Trigger: Token expiry sweep (removeExpiredTokens) or cancelToken hitting ConnectionLoss, SessionExpired, NoAuth or BADACL on the delete(); also checkExists() against a deleted tokens root.

Common situations: Extended ZK partition overlapping the expiry scan interval; session expiry from long GC or network blackholes; ACL drift after principal rotation; ZKDTSMTokensRoot removed while services run.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ebd6b091860188e8. Report an issue: GitHub.