apache/hadoop · error · IOException

Tokens cannot be fetched from path {TOKEN_PATH}

Error message

Tokens cannot be fetched from path {TOKEN_PATH}

What it means

ZKDelegationTokenSecretManagerImpl.rebuildTokenCache rebuilds the local token cache by listing all delegation-token znodes under TOKEN_PATH with the raw ZooKeeper client (Curator's sorting wrapper is too slow for millions of tokens). Any KeeperException or InterruptedException from getChildren is rethrown as IOException('Tokens cannot be fetched from path ' + TOKEN_PATH). rebuildTokenCache runs at initialization and on cache refreshes, so ZooKeeper problems surface here early.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/security/token/ZKDelegationTokenSecretManagerImpl.java:157

  /**
   * This function will rebuild local token cache from zk storage.
   * It is first called when the secret manager is initialized and
   * then regularly at a configured interval.
   *
   * @param initial whether this is called during initialization
   * @throws IOException
   */
  private void rebuildTokenCache(boolean initial) throws IOException {
    localTokenCache.clear();
    // Use bare zookeeper client to get all children since curator will
    // wrap the same API with a sorting process. This is time consuming given
    // millions of tokens
    List<String> zkTokens;
    try {
      zkTokens = getZooKeeperClient().getChildren(TOKEN_PATH, false);
    } catch (KeeperException | InterruptedException e) {
      throw new IOException("Tokens cannot be fetched from path "
          + TOKEN_PATH, e);
    }
    byte[] data;
    for (String tokenPath : zkTokens) {
      try {
        data = zkClient.getData().forPath(
            ZK_DTSM_TOKENS_ROOT + "/" + tokenPath);
      } catch (KeeperException.NoNodeException e) {
        LOG.debug("No node in path [" + tokenPath + "]");
        continue;
      } catch (Exception ex) {
        throw new IOException(ex);
      }
      // Store data to currentTokenMap
      AbstractDelegationTokenIdentifier ident = processTokenAddOrUpdate(data);
      // Store data to localTokenCache for sync
      localTokenCache.add(ident);
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Check the router log just before this exception for Curator connection-state events identifying the ZK failure mode.
2. Verify ZK reachability and the znode from the router host: zkCli.sh -server <zk> then ls on the token root path.
3. If NoNode on a fresh ensemble, ensure the ZK secret manager initialization creates the root (starting one correctly configured router normally does), or create the parent znode manually.
4. Align ZK auth: configure the same digest/sasl scheme and ACLs the token store was created with.
5. Restart the router once ZK is healthy if the failure happened during startup.

try {
  secretManager.startThreads(); // triggers rebuildTokenCache
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Tokens cannot be fetched")
      && e.getCause() instanceof KeeperException) {
    KeeperException ke = (KeeperException) e.getCause();
    if (ke.code() == KeeperException.Code.CONNECTIONLOSS
        || ke.code() == KeeperException.Code.SESSIONEXPIRED) {
      // transient: wait for ZK recovery and retry rebuild/start once
    } else {
      throw e; // NoNode/NoAuth = provisioning/ACL problem, retrying won't help
    }
  } else {
    throw e;
  }
}

When it happens

Trigger: KeeperException.ConnectionLoss/SessionExpired while listing tokens; KeeperException.NoNode when the tokens parent znode does not exist yet; KeeperException.NoAuth when ZK ACLs deny the router's credentials; InterruptedException during manager shutdown racing a rebuild.

Common situations: ZooKeeper quorum down or network-partitioned from the router; ZK auth (digest/sasl) not configured on the router though the znodes are protected; first use on a fresh ensemble where the token root was never created; ZK session flaps under load with millions of token znodes.

Related errors

• Zookeeper client is null
• Failed to create SecretManager
• Delegation Token can be issued only with kerberos or web aut
• Delegation Token can be renewed only with kerberos or web au
• Counter table not initialized: {table}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/d8b84227ddfd5c79. Report an issue: GitHub.