apache/hadoop · warning · IOException

Cannot get access token since BlockKeyUpdater is not running

Error message

Cannot get access token since BlockKeyUpdater is not running

What it means

KeyManager.getAccessToken issues block tokens for balancer moves when block access tokens are enabled (dfs.block.access.token.enable). If the internal shouldRun flag is false it refuses with this IOException. shouldRun flips false in two places: KeyManager.close() (balancer/NameNodeConnector shutting down) and when the BlockKeyUpdater daemon dies from an unexpected Throwable after logging 'Exception in block key updater thread'.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/KeyManager.java:110

      this.blockTokenSecretManager = null;
      this.blockKeyUpdater = null;
    }
  }
  
  public void startBlockKeyUpdater() {
    if (blockKeyUpdater != null) {
      blockKeyUpdater.daemon.start();
    }
  }

  /** Get an access token for a block. */
  public Token<BlockTokenIdentifier> getAccessToken(ExtendedBlock eb,
      StorageType[] storageTypes, String[] storageIds) throws IOException {
    if (!isBlockTokenEnabled) {
      return BlockTokenSecretManager.DUMMY_TOKEN;
    } else {
      if (!shouldRun) {
        throw new IOException(
            "Cannot get access token since BlockKeyUpdater is not running");
      }
      return blockTokenSecretManager.generateToken(null, eb,
          EnumSet.of(BlockTokenIdentifier.AccessMode.REPLACE,
              BlockTokenIdentifier.AccessMode.COPY), storageTypes, storageIds);
    }
  }

  @Override
  public DataEncryptionKey newDataEncryptionKey() {
    if (encryptDataTransfer) {
      synchronized (this) {
        if (encryptionKey == null ||
            encryptionKey.expiryDate < timer.now()) {
          // Encryption Key (EK) is generated from Block Key (BK).
          // Check if EK is expired, and generate a new one using the current BK
          // if so, otherwise continue to use the previously generated EK.
          //

View on GitHub (pinned to 2add963021)

Solutions

If it appears only at balancer shutdown, ignore it - pending moves are dropped and the next run re-plans them
Check the balancer log just above for 'Exception in block key updater thread' or 'Failed to set keys' to find the root cause (usually NameNode connectivity)
Fix NameNode reachability/HA failover behavior and rerun the balancer

Defensive patterns

Strategy: try-catch

Try / catch

try { token = nnc.getKeyManager().getAccessToken(eb, storageTypes, storageIds); }
catch (IOException e) {
  if (e.getMessage().contains("BlockKeyUpdater is not running")) { LOG.warn("Key manager stopped; aborting remaining moves"); abortMoves(); }
  else throw e;
}

Prevention

Close NameNodeConnector only after dispatcher threads have drained (join move threads before nnc.close())
Watch for 'Exception in block key updater thread' in logs - it is the root cause precursor to this error
On embedded use (e.g. router balancer), tolerate this error during shutdown instead of failing the run summary

When it happens

Trigger: A dispatch thread requests a token after the balancer run finished and close() raced ahead of pending moves; or the BlockKeyUpdater thread crashed (e.g. persistent NameNode RPC failures in namenode.getBlockKeys()) so shouldRun was set false while moves were still in flight.

Common situations: Benign race at the end of a balancer run (moves aborted as the tool exits); NameNode unreachable or restarting while the balancer was mid-run; long-running balancer sessions outliving key-update connectivity.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/849d68fd96dbb610. Report an issue: GitHub.