apache/hadoop · error · IOException

Can't add persisted delegation token to a running SecretMana

Error message

Can't add persisted delegation token to a running SecretManager.

What it means

DelegationTokenSecretManager.addPersistedDelegationToken throws IOException when it is asked to insert a token replayed from fsimage/edit logs while the manager is already running. Persisted-token insertion is an edit-log-replay operation reserved for the loading phase (running == false); once the manager serves RPCs, new tokens must arrive via requestNewPassword/issue, not replay. The guard prevents replay code from mutating live token state.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java:280

    }

    return new SecretManagerState(s, keys, tokens);
  }

  /**
   * This method is intended to be used only while reading edit logs.
   * 
   * @param identifier DelegationTokenIdentifier read from the edit logs or
   * fsimage
   * 
   * @param expiryTime token expiry time
   * @throws IOException
   */
  public synchronized void addPersistedDelegationToken(
      DelegationTokenIdentifier identifier, long expiryTime) throws IOException {
    if (running) {
      // a safety check
      throw new IOException(
          "Can't add persisted delegation token to a running SecretManager.");
    }
    int keyId = identifier.getMasterKeyId();
    DelegationKey dKey = allKeys.get(keyId);
    if (dKey == null) {
      LOG
          .warn("No KEY found for persisted identifier "
              + identifier.toString());
      return;
    }
    byte[] password = createPassword(identifier.getBytes(), dKey.getKey());
    if (identifier.getSequenceNumber() > this.delegationTokenSequenceNumber) {
      this.delegationTokenSequenceNumber = identifier.getSequenceNumber();
    }
    if (currentTokens.get(identifier) == null) {
      currentTokens.put(identifier, new DelegationTokenInformation(expiryTime,
          password, getTrackingIdIfEnabled(identifier)));
    } else {

View on GitHub (pinned to 2add963021)

Solutions

1. Fix startup ordering: complete fsimage load AND edit-log replay (including OP_DELEGATION_TOKEN entries) before startThreads()/activate the RPC server.
2. For programs replaying logs, instantiate a dedicated non-running DelegationTokenSecretManager for replay and only then publish its state.
3. In tests, call manager.stopThreads() or use a fresh manager before replaying persisted tokens.
4. Check for duplicate start paths (e.g., an extra manual start()) in custom NameNode bootstrap code.

Example fix

// before
dtSecretManager.startThreads();
editLog.recoverUncloseSegments(); // replay calls addPersistedDelegationToken -> IOException

// after
// replay everything first, then start
editLog.recoverUncloseSegments(); // addPersistedDelegationToken ok (not running)
dtSecretManager.startThreads();

assert !dtSecretManager.isRunning() : "replay must precede startThreads()";
dtSecretManager.addPersistedDelegationToken(identifier, expiryTime);

try {
  dtSecretManager.addPersistedDelegationToken(identifier, expiryTime);
} catch (IOException e) {
  if (e.getMessage().contains("running SecretManager")) {
    // replay hit a live manager: replay into a dedicated non-running instance
  } else { throw e; }
}

When it happens

Trigger: addPersistedDelegationToken(identifier, expiryTime) is called after startThreads() — e.g., edit-log replay continues after the secret manager was started, or replay logic is applied to a live NN's manager.

Common situations: NameNode startup ordering broken so that the RPC server/secret manager activates before edit-log replay finishes; tools that replay edit logs against a running NN's secret manager; tests that start the manager then feed it log-derived tokens.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Can't load state from image in a running SecretManager.
• Can't update persisted delegation token renewal to a running
• Fetch of delegation token failed
• currentKey hasn't been initialized.
• No delegation token found for this identifier

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/fd63f7f1b2ced032. Report an issue: GitHub.