{"record":{"id":"fd63f7f1b2ced032","repo":"apache/hadoop","slug":"can-t-add-persisted-delegation-token-to-a-running-fd63f7","errorCode":null,"errorMessage":"Can't add persisted delegation token to a running SecretManager.","messagePattern":"Can't add persisted delegation token to a running SecretManager\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java","lineNumber":280,"sourceCode":"    }\n\n    return new SecretManagerState(s, keys, tokens);\n  }\n\n  /**\n   * This method is intended to be used only while reading edit logs.\n   * \n   * @param identifier DelegationTokenIdentifier read from the edit logs or\n   * fsimage\n   * \n   * @param expiryTime token expiry time\n   * @throws IOException\n   */\n  public synchronized void addPersistedDelegationToken(\n      DelegationTokenIdentifier identifier, long expiryTime) throws IOException {\n    if (running) {\n      // a safety check\n      throw new IOException(\n          \"Can't add persisted delegation token to a running SecretManager.\");\n    }\n    int keyId = identifier.getMasterKeyId();\n    DelegationKey dKey = allKeys.get(keyId);\n    if (dKey == null) {\n      LOG\n          .warn(\"No KEY found for persisted identifier \"\n              + identifier.toString());\n      return;\n    }\n    byte[] password = createPassword(identifier.getBytes(), dKey.getKey());\n    if (identifier.getSequenceNumber() > this.delegationTokenSequenceNumber) {\n      this.delegationTokenSequenceNumber = identifier.getSequenceNumber();\n    }\n    if (currentTokens.get(identifier) == null) {\n      currentTokens.put(identifier, new DelegationTokenInformation(expiryTime,\n          password, getTrackingIdIfEnabled(identifier)));\n    } else {","sourceCodeStart":262,"sourceCodeEnd":298,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java#L262-L298","documentation":"DelegationTokenSecretManager.addPersistedDelegationToken throws IOException when it is asked to insert a token replayed from fsimage/edit logs while the manager is already running. Persisted-token insertion is an edit-log-replay operation reserved for the loading phase (running == false); once the manager serves RPCs, new tokens must arrive via requestNewPassword/issue, not replay. The guard prevents replay code from mutating live token state.","triggerScenarios":"addPersistedDelegationToken(identifier, expiryTime) is called after startThreads() — e.g., edit-log replay continues after the secret manager was started, or replay logic is applied to a live NN's manager.","commonSituations":"NameNode startup ordering broken so that the RPC server/secret manager activates before edit-log replay finishes; tools that replay edit logs against a running NN's secret manager; tests that start the manager then feed it log-derived tokens.","solutions":["Fix startup ordering: complete fsimage load AND edit-log replay (including OP_DELEGATION_TOKEN entries) before startThreads()/activate the RPC server.","For programs replaying logs, instantiate a dedicated non-running DelegationTokenSecretManager for replay and only then publish its state.","In tests, call manager.stopThreads() or use a fresh manager before replaying persisted tokens.","Check for duplicate start paths (e.g., an extra manual start()) in custom NameNode bootstrap code."],"exampleFix":"// before\ndtSecretManager.startThreads();\neditLog.recoverUncloseSegments(); // replay calls addPersistedDelegationToken -> IOException\n\n// after\n// replay everything first, then start\neditLog.recoverUncloseSegments(); // addPersistedDelegationToken ok (not running)\ndtSecretManager.startThreads();","handlingStrategy":"validation","validationCode":"assert !dtSecretManager.isRunning() : \"replay must precede startThreads()\";\ndtSecretManager.addPersistedDelegationToken(identifier, expiryTime);","typeGuard":null,"tryCatchPattern":"try {\n  dtSecretManager.addPersistedDelegationToken(identifier, expiryTime);\n} catch (IOException e) {\n  if (e.getMessage().contains(\"running SecretManager\")) {\n    // replay hit a live manager: replay into a dedicated non-running instance\n  } else { throw e; }\n}","preventionTips":["Finish fsimage load and edit-log replay before activating the RPC server/secret manager.","In replay tooling, use a separate non-running manager instance.","Reset manager state between replay passes in tests."],"tags":["hdfs","delegation-token","security","edit-log","lifecycle"],"backgroundTag":"invalid-state-transition","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}