apache/hadoop · error · IOException

Can't update persisted delegation token renewal to a running

Error message

Can't update persisted delegation token renewal to a running SecretManager.

What it means

DelegationTokenSecretManager.updatePersistedTokenRenewal throws IOException when an OP_DELEGATION_TOKEN_RENEW record from edit logs is applied while the secret manager is already running. Renewal replay is part of image/log loading; a running manager must only see renewals through the live renewToken RPC path. The check (running == true) catches startup ordering bugs where replay happens after the manager went live.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java:326

   * @throws IOException
   */
  public synchronized void updatePersistedMasterKey(DelegationKey key)
      throws IOException {
    addKey(key);
  }
  
  /**
   * Update the token cache with renewal record in edit logs.
   * 
   * @param identifier DelegationTokenIdentifier of the renewed token
   * @param expiryTime expirty time in milliseconds
   * @throws IOException
   */
  public synchronized void updatePersistedTokenRenewal(
      DelegationTokenIdentifier identifier, long expiryTime) throws IOException {
    if (running) {
      // a safety check
      throw new IOException(
          "Can't update persisted delegation token renewal to a running SecretManager.");
    }
    DelegationTokenInformation info = null;
    info = currentTokens.get(identifier);
    if (info != null) {
      int keyId = identifier.getMasterKeyId();
      byte[] password = createPassword(identifier.getBytes(), allKeys
          .get(keyId).getKey());
      currentTokens.put(identifier, new DelegationTokenInformation(expiryTime,
          password, getTrackingIdIfEnabled(identifier)));
    }
  }

  /**
   *  Update the token cache with the cancel record in edit logs
   *  
   *  @param identifier DelegationTokenIdentifier of the canceled token
   *  @throws IOException

View on GitHub (pinned to 2add963021)

Solutions

1. Reorder startup: finish fsimage load and full edit-log replay before starting the secret manager threads / RPC server.
2. Use a separate, non-started DelegationTokenSecretManager for any offline replay of renew records.
3. In tests, create a fresh manager per replay pass; never reuse a started one.
4. Audit for double start of the manager (extra startThreads call) in custom bootstrap code.

Example fix

// before
dtSecretManager.startThreads();
replayEditLogOp(OP_DELEGATION_TOKEN_RENEW, id, expiry); // IOException

// after
replayEditLogOp(OP_DELEGATION_TOKEN_RENEW, id, expiry); // manager not yet running
dtSecretManager.startThreads();

assert !dtSecretManager.isRunning() : "renewal replay must precede startThreads()";
dtSecretManager.updatePersistedTokenRenewal(identifier, expiryTime);

try {
  dtSecretManager.updatePersistedTokenRenewal(identifier, expiryTime);
} catch (IOException e) {
  if (e.getMessage().contains("running SecretManager")) {
    // reorder: replay edits first, then start manager; or replay into fresh instance
  } else { throw e; }
}

When it happens

Trigger: updatePersistedTokenRenewal(identifier, expiryTime) is invoked after startThreads(), i.e., edit-log replay (or a tool re-applying renew records) runs against a started secret manager.

Common situations: NameNode startup enabling the RPC server before replay completes; custom log-replay tooling pointed at a live manager; tests that start the manager then replay logs; double-started secret manager instances.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Can't add persisted delegation token to a running SecretMana
• Same delegation token being added twice; invalid entry in fs
• Could not start ${secretManagerClass}: ${ex.toString()}
• Can't load state from image in a running SecretManager.
• Error replaying edit log at offset {}. Expected transaction

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a967cc5ee056587b. Report an issue: GitHub.