apache/hadoop · error · IOException

Can't load state from image in a running SecretManager.

Error message

Can't load state from image in a running SecretManager.

What it means

DelegationTokenSecretManager.loadSecretManagerStateCompat throws IOException when asked to deserialize token/key state from fsimage while the manager is already running (running == true). Loading persisted state initializes currentTokens/allKeys and must happen before startThreads() turns the manager on; afterwards, mutating from an image would clobber live state. This is a lifecycle guard against double-load or loading into a live secret manager.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java:174

    DelegationTokenInformation info = currentTokens.get(dtId);
    if (info != null) {
      return info.getRenewDate();
    } else {
      throw new IOException("No delegation token found for this identifier");
    }
  }

  /**
   * Load SecretManager state from fsimage.
   * 
   * @param in input stream to read fsimage
   * @throws IOException
   */
  public synchronized void loadSecretManagerStateCompat(DataInput in)
      throws IOException {
    if (running) {
      // a safety check
      throw new IOException(
          "Can't load state from image in a running SecretManager.");
    }
    serializerCompat.load(in);
  }

  public static class SecretManagerState {
    public final SecretManagerSection section;
    public final List<SecretManagerSection.DelegationKey> keys;
    public final List<SecretManagerSection.PersistToken> tokens;

    public SecretManagerState(
        SecretManagerSection s,
        List<SecretManagerSection.DelegationKey> keys,
        List<SecretManagerSection.PersistToken> tokens) {
      this.section = s;
      this.keys = keys;
      this.tokens = tokens;
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Load all fsimage state before starting the secret manager: call loadSecretManagerState* during image load, then startThreads()/start() exactly once.
2. If a second image must be processed, construct a new DelegationTokenSecretManager instance for the load (stop the old one first via stopThreads()).
3. In tests, reset the manager (create a fresh instance) between image loads instead of reusing one.
4. Audit custom FSImage consumers for accidental double invocation of the load path after startup.

Example fix

// before
dtSecretManager.startThreads();
... 
dtSecretManager.loadSecretManagerStateCompat(in); // IOException: running SecretManager

// after
// load first, start once
dtSecretManager.loadSecretManagerStateCompat(in);
dtSecretManager.startThreads();

// Guard the load path: only load when not running
if (dtSecretManager.isRunning()) {
  throw new IllegalStateException("Refusing image load: use a fresh secret manager");
}
dtSecretManager.loadSecretManagerStateCompat(in);

try {
  dtSecretManager.loadSecretManagerStateCompat(in);
} catch (IOException e) {
  if (e.getMessage().contains("running SecretManager")) {
    // lifecycle bug: create a new manager instance for this load
    dtSecretManager = new DelegationTokenSecretManager(...);
    dtSecretManager.loadSecretManagerStateCompat(in);
  } else { throw e; }
}

When it happens

Trigger: loadSecretManagerStateCompat(in) is invoked after start() — e.g., a NameNode (or a tool embedding the NN image loader) attempts to load a second fsimage, or code loads checkpoint state after the token manager already started its renewal/expiry threads.

Common situations: Custom tooling/tests that call FSImageFormat loadDelegateSections twice; NN rolling upgrade or checkpoint reload path that reuses a started DelegationTokenSecretManager; embedding HDFS image loading in an application that also runs a secret manager; framework code that loads images on a schedule.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Can't add persisted delegation token to a running SecretMana
• Fetch of delegation token failed
• currentKey hasn't been initialized.
• No delegation token found for this identifier
• Same delegation token being added twice; invalid entry in fs

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f471cf3eb726ef3f. Report an issue: GitHub.