{"record":{"id":"d8b84227ddfd5c79","repo":"apache/hadoop","slug":"tokens-cannot-be-fetched-from-path-token-path","errorCode":null,"errorMessage":"Tokens cannot be fetched from path {TOKEN_PATH}","messagePattern":"Tokens cannot be fetched from path (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/security/token/ZKDelegationTokenSecretManagerImpl.java","lineNumber":157,"sourceCode":"\n  /**\n   * This function will rebuild local token cache from zk storage.\n   * It is first called when the secret manager is initialized and\n   * then regularly at a configured interval.\n   *\n   * @param initial whether this is called during initialization\n   * @throws IOException\n   */\n  private void rebuildTokenCache(boolean initial) throws IOException {\n    localTokenCache.clear();\n    // Use bare zookeeper client to get all children since curator will\n    // wrap the same API with a sorting process. This is time consuming given\n    // millions of tokens\n    List<String> zkTokens;\n    try {\n      zkTokens = getZooKeeperClient().getChildren(TOKEN_PATH, false);\n    } catch (KeeperException | InterruptedException e) {\n      throw new IOException(\"Tokens cannot be fetched from path \"\n          + TOKEN_PATH, e);\n    }\n    byte[] data;\n    for (String tokenPath : zkTokens) {\n      try {\n        data = zkClient.getData().forPath(\n            ZK_DTSM_TOKENS_ROOT + \"/\" + tokenPath);\n      } catch (KeeperException.NoNodeException e) {\n        LOG.debug(\"No node in path [\" + tokenPath + \"]\");\n        continue;\n      } catch (Exception ex) {\n        throw new IOException(ex);\n      }\n      // Store data to currentTokenMap\n      AbstractDelegationTokenIdentifier ident = processTokenAddOrUpdate(data);\n      // Store data to localTokenCache for sync\n      localTokenCache.add(ident);\n    }","sourceCodeStart":139,"sourceCodeEnd":175,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/security/token/ZKDelegationTokenSecretManagerImpl.java#L139-L175","documentation":"ZKDelegationTokenSecretManagerImpl.rebuildTokenCache rebuilds the local token cache by listing all delegation-token znodes under TOKEN_PATH with the raw ZooKeeper client (Curator's sorting wrapper is too slow for millions of tokens). Any KeeperException or InterruptedException from getChildren is rethrown as IOException('Tokens cannot be fetched from path ' + TOKEN_PATH). rebuildTokenCache runs at initialization and on cache refreshes, so ZooKeeper problems surface here early.","triggerScenarios":"KeeperException.ConnectionLoss/SessionExpired while listing tokens; KeeperException.NoNode when the tokens parent znode does not exist yet; KeeperException.NoAuth when ZK ACLs deny the router's credentials; InterruptedException during manager shutdown racing a rebuild.","commonSituations":"ZooKeeper quorum down or network-partitioned from the router; ZK auth (digest/sasl) not configured on the router though the znodes are protected; first use on a fresh ensemble where the token root was never created; ZK session flaps under load with millions of token znodes.","solutions":["Check the router log just before this exception for Curator connection-state events identifying the ZK failure mode.","Verify ZK reachability and the znode from the router host: zkCli.sh -server <zk> then ls on the token root path.","If NoNode on a fresh ensemble, ensure the ZK secret manager initialization creates the root (starting one correctly configured router normally does), or create the parent znode manually.","Align ZK auth: configure the same digest/sasl scheme and ACLs the token store was created with.","Restart the router once ZK is healthy if the failure happened during startup."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  secretManager.startThreads(); // triggers rebuildTokenCache\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"Tokens cannot be fetched\")\n      && e.getCause() instanceof KeeperException) {\n    KeeperException ke = (KeeperException) e.getCause();\n    if (ke.code() == KeeperException.Code.CONNECTIONLOSS\n        || ke.code() == KeeperException.Code.SESSIONEXPIRED) {\n      // transient: wait for ZK recovery and retry rebuild/start once\n    } else {\n      throw e; // NoNode/NoAuth = provisioning/ACL problem, retrying won't help\n    }\n  } else {\n    throw e;\n  }\n}","preventionTips":["Verify ZK ensemble reachability and the token root znode from the router before enabling the ZK token driver.","Configure ZK auth/ACLs to match how the token znodes were created; mismatches show up as NoAuth on this path.","Alert on Curator connection-state transitions so ZK flaps are handled before token cache rebuilds fail."],"tags":["hdfs","router-based-federation","delegation-token","zookeeper","distributed-cache"],"backgroundTag":"zookeeper-connection-failure","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}