{"record":{"id":"849d68fd96dbb610","repo":"apache/hadoop","slug":"cannot-get-access-token-since-blockkeyupdater-is-n","errorCode":null,"errorMessage":"Cannot get access token since BlockKeyUpdater is not running","messagePattern":"Cannot get access token since BlockKeyUpdater is not running","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"warning","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/KeyManager.java","lineNumber":110,"sourceCode":"      this.blockTokenSecretManager = null;\n      this.blockKeyUpdater = null;\n    }\n  }\n  \n  public void startBlockKeyUpdater() {\n    if (blockKeyUpdater != null) {\n      blockKeyUpdater.daemon.start();\n    }\n  }\n\n  /** Get an access token for a block. */\n  public Token<BlockTokenIdentifier> getAccessToken(ExtendedBlock eb,\n      StorageType[] storageTypes, String[] storageIds) throws IOException {\n    if (!isBlockTokenEnabled) {\n      return BlockTokenSecretManager.DUMMY_TOKEN;\n    } else {\n      if (!shouldRun) {\n        throw new IOException(\n            \"Cannot get access token since BlockKeyUpdater is not running\");\n      }\n      return blockTokenSecretManager.generateToken(null, eb,\n          EnumSet.of(BlockTokenIdentifier.AccessMode.REPLACE,\n              BlockTokenIdentifier.AccessMode.COPY), storageTypes, storageIds);\n    }\n  }\n\n  @Override\n  public DataEncryptionKey newDataEncryptionKey() {\n    if (encryptDataTransfer) {\n      synchronized (this) {\n        if (encryptionKey == null ||\n            encryptionKey.expiryDate < timer.now()) {\n          // Encryption Key (EK) is generated from Block Key (BK).\n          // Check if EK is expired, and generate a new one using the current BK\n          // if so, otherwise continue to use the previously generated EK.\n          //","sourceCodeStart":92,"sourceCodeEnd":128,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/KeyManager.java#L92-L128","documentation":"KeyManager.getAccessToken issues block tokens for balancer moves when block access tokens are enabled (dfs.block.access.token.enable). If the internal shouldRun flag is false it refuses with this IOException. shouldRun flips false in two places: KeyManager.close() (balancer/NameNodeConnector shutting down) and when the BlockKeyUpdater daemon dies from an unexpected Throwable after logging 'Exception in block key updater thread'.","triggerScenarios":"A dispatch thread requests a token after the balancer run finished and close() raced ahead of pending moves; or the BlockKeyUpdater thread crashed (e.g. persistent NameNode RPC failures in namenode.getBlockKeys()) so shouldRun was set false while moves were still in flight.","commonSituations":"Benign race at the end of a balancer run (moves aborted as the tool exits); NameNode unreachable or restarting while the balancer was mid-run; long-running balancer sessions outliving key-update connectivity.","solutions":["If it appears only at balancer shutdown, ignore it - pending moves are dropped and the next run re-plans them","Check the balancer log just above for 'Exception in block key updater thread' or 'Failed to set keys' to find the root cause (usually NameNode connectivity)","Fix NameNode reachability/HA failover behavior and rerun the balancer"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try { token = nnc.getKeyManager().getAccessToken(eb, storageTypes, storageIds); }\ncatch (IOException e) {\n  if (e.getMessage().contains(\"BlockKeyUpdater is not running\")) { LOG.warn(\"Key manager stopped; aborting remaining moves\"); abortMoves(); }\n  else throw e;\n}","preventionTips":["Close NameNodeConnector only after dispatcher threads have drained (join move threads before nnc.close())","Watch for 'Exception in block key updater thread' in logs - it is the root cause precursor to this error","On embedded use (e.g. router balancer), tolerate this error during shutdown instead of failing the run summary"],"tags":["hdfs","balancer","security","block-token","shutdown-race"],"backgroundTag":"component-shutdown-race","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}