apache/hadoop · critical · IOException

${nodePath} znode could not be created !!

Error message

${nodePath} znode could not be created !!

What it means

ZKDelegationTokenSecretManager persists HA delegation-token state in ZooKeeper. During startThreads() it creates the persistent parent znodes under its namespace (/zkdtsm/ZKDTSMRoot/..., e.g. ZKDTSMTokensRoot, ZKDTSMMasterKeyRoot) via createPersistentNode(). KeeperException.NodeExistsException is tolerated (idempotent create), but every other failure - connection loss, session expiry, NoAuth, bad ACL - is rethrown as IOException "<path> znode could not be created !!". This aborts service startup for whatever HA service (NameNode ZKFC, ResourceManager, WebHDFS) owns the token manager.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java:481

    } catch (Exception e) {
      LOG.error("Could not stop KeyCache", e);
    }
    try {
      if (!isExternalClient && (zkClient != null)) {
        zkClient.close();
      }
    } catch (Exception e) {
      LOG.error("Could not stop Curator Framework", e);
    }
  }

  private void createPersistentNode(String nodePath) throws Exception {
    try {
      zkClient.create().withMode(CreateMode.PERSISTENT).forPath(nodePath);
    } catch (KeeperException.NodeExistsException ne) {
      LOG.debug(nodePath + " znode already exists !!");
    } catch (Exception e) {
      throw new IOException(nodePath + " znode could not be created !!", e);
    }
  }

  @Override
  protected int getDelegationTokenSeqNum() {
    return delTokSeqCounter.getCount();
  }

  private int incrSharedCount(SharedCount sharedCount, int batchSize)
      throws Exception {
    while (true) {
      // Loop until we successfully increment the counter
      VersionedValue<Integer> versionedValue = sharedCount.getVersionedValue();
      if (sharedCount.trySetCount(
          versionedValue, versionedValue.getValue() + batchSize)) {
        return versionedValue.getValue();
      }
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Verify ZK reachability and permissions from the service host: zkCli.sh -server <quorum> ls /zkdtsm/ZKDTSMRoot (create/read/write for the service principal).
2. Check zk-dt-secret-manager.zkConnectionString and, for secured ZK, zk-dt-secret-manager.zkAuthType=digest|kerberos plus kerberos.keytab/kerberos.principal (or the JAAS Client login entry) match the ZK server's SASL config.
3. If stale ACLs block creation, stop services and delete the /zkdtsm working-path tree with zkCli so it is recreated with the current principal.
4. For flaky links raise zk-dt-secret-manager.zkNumRetries and zk-dt-secret-manager.zkSessionTimeout, then restart the service.

Example fix

<!-- before: secured HA ZK with no auth config; create() fails NoAuth/SessionExpired -->
<property>
  <name>zk-dt-secret-manager.zkConnectionString</name>
  <value>zk1:2181,zk2:2181,zk3:2181</value>
</property>
<!-- after: add auth matching the ZK server -->
<property>
  <name>zk-dt-secret-manager.zkAuthType</name>
  <value>kerberos</value>
</property>
<property>
  <name>zk-dt-secret-manager.kerberos.principal</name>
  <value>nn/_HOST@EXAMPLE.COM</value>
</property>
<property>
  <name>zk-dt-secret-manager.kerberos.keytab</name>
  <value>/etc/security/keytabs/nn.service.keytab</value>
</property>

// Preflight before starting the token secret manager:
// the service principal must be able to create /zkdtsm/ZKDTSMRoot children.
try (CuratorFramework zk = CuratorFrameworkFactory.newClient(
        quorum, new RetryNTimes(1, 1000))) {
  zk.start();
  zk.create().creatingParentsIfNeeded()
     .withACL(ZooDefs.Ids.OPEN_ACL_UNSAFE) // match your production ACL policy
     .forPath("/zkdtsm/ZKDTSMRoot");
  zk.delete().forPath("/zkdtsm/ZKDTSMRoot"); // round-trip create/delete proves perms
}

try {
  zkSecretManager.startThreads();
} catch (IOException e) {
  Throwable cause = (e.getCause() instanceof KeeperException) ? e.getCause() : e;
  if (cause instanceof KeeperException.NoAuthException) {
    // fix ACLs/JAAS before retrying; retry will not help
  } else if (cause instanceof KeeperException.ConnectionLoss) {
    // ZK unreachable: retry after restoring connectivity
  }
}

When it happens

Trigger: startThreads() -> createPersistentNode(ZK_DTSM_TOKENS_ROOT / ZK_DTSM_MASTER_KEY_ROOT / ZK_DTSM_SEQNUM_ROOT) when Curator create().forPath() fails with anything except NodeExists: ZK ensemble unreachable (bad zk-dt-secret-manager.zkConnectionString), SASL/Kerberos auth failure (zkAuthType/kerberos.* mismatch, missing JAAS Client section), znode ACL owned by a different principal from a previous deployment, connection/session timeout exhausted after zkNumRetries.

Common situations: First secured HA deployment without matching ZK JAAS config; /zkdtsm tree left with CREATOR_ALL_ACL by an earlier run under a different service user; typo in the ZK connect string; ZK quorum loss during HA failover; SSL-enabled ZK (zk-dt-secret-manager.ssl.*) with missing keystore/truststore config.

Related errors

• Could not increment shared counter !!
• Could not update Stored Token ZKDTSMDelegationToken_${sequen
• Could not remove Stored Token ZKDTSMDelegationToken_${sequen
• Could not start ${secretManagerClass}: ${ex.toString()}
• Failed to create SecretManager

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/07481364c8f7e84e. Report an issue: GitHub.