apache/hadoop · critical · IOException

znode could not be created !!

Error message

${nodePath} znode could not be created !!

What it means

ZKDelegationTokenSecretManager persists HA delegation-token state in ZooKeeper. During startThreads() it creates the persistent parent znodes under its namespace (/zkdtsm/ZKDTSMRoot/..., e.g. ZKDTSMTokensRoot, ZKDTSMMasterKeyRoot) via createPersistentNode(). KeeperException.NodeExistsException is tolerated (idempotent create), but every other failure - connection loss, session expiry, NoAuth, bad ACL - is rethrown as IOException "<path> znode could not be created !!". This aborts service startup for whatever HA service (NameNode ZKFC, ResourceManager, WebHDFS) owns the token manager.

Solutions

1. Verify ZK reachability and permissions from the service host: zkCli.sh -server <quorum> ls /zkdtsm/ZKDTSMRoot (create/read/write for the service principal).
2. Check zk-dt-secret-manager.zkConnectionString and, for secured ZK, zk-dt-secret-manager.zkAuthType=digest|kerberos plus kerberos.keytab/kerberos.principal (or the JAAS Client login entry) match the ZK server's SASL config.
3. If stale ACLs block creation, stop services and delete the /zkdtsm working-path tree with zkCli so it is recreated with the current principal.
4. For flaky links raise zk-dt-secret-manager.zkNumRetries and zk-dt-secret-manager.zkSessionTimeout, then restart the service.

Example fix

<!-- before: secured HA ZK with no auth config; create() fails NoAuth/SessionExpired -->
<property>
  <name>zk-dt-secret-manager.zkConnectionString</name>
  <value>zk1:2181,zk2:2181,zk3:2181</value>
</property>
<!-- after: add auth matching the ZK server -->
<property>
  <name>zk-dt-secret-manager.zkAuthType</name>
  <value>kerberos</value>
</property>
<property>
  <name>zk-dt-secret-manager.kerberos.principal</name>
  <value>nn/_HOST@EXAMPLE.COM</value>
</property>
<property>
  <name>zk-dt-secret-manager.kerberos.keytab</name>
  <value>/etc/security/keytabs/nn.service.keytab</value>
</property>

// Preflight before starting the token secret manager:
// the service principal must be able to create /zkdtsm/ZKDTSMRoot children.
try (CuratorFramework zk = CuratorFrameworkFactory.newClient(
        quorum, new RetryNTimes(1, 1000))) {
  zk.start();
  zk.create().creatingParentsIfNeeded()
     .withACL(ZooDefs.Ids.OPEN_ACL_UNSAFE) // match your production ACL policy
     .forPath("/zkdtsm/ZKDTSMRoot");
  zk.delete().forPath("/zkdtsm/ZKDTSMRoot"); // round-trip create/delete proves perms
}

try {
  zkSecretManager.startThreads();
} catch (IOException e) {
  Throwable cause = (e.getCause() instanceof KeeperException) ? e.getCause() : e;
  if (cause instanceof KeeperException.NoAuthException) {
    // fix ACLs/JAAS before retrying; retry will not help
  } else if (cause instanceof KeeperException.ConnectionLoss) {
    // ZK unreachable: retry after restoring connectivity
  }
}

When it happens

Trigger: startThreads() -> createPersistentNode(ZK_DTSM_TOKENS_ROOT / ZK_DTSM_MASTER_KEY_ROOT / ZK_DTSM_SEQNUM_ROOT) when Curator create().forPath() fails with anything except NodeExists: ZK ensemble unreachable (bad zk-dt-secret-manager.zkConnectionString), SASL/Kerberos auth failure (zkAuthType/kerberos.* mismatch, missing JAAS Client section), znode ACL owned by a different principal from a previous deployment, connection/session timeout exhausted after zkNumRetries.

Common situations: First secured HA deployment without matching ZK JAAS config; /zkdtsm tree left with CREATOR_ALL_ACL by an earlier run under a different service user; typo in the ZK connect string; ZK quorum loss during HA failover; SSL-enabled ZK (zk-dt-secret-manager.ssl.*) with missing keystore/truststore config.

Related errors

• Could not increment shared counter !!
• Could not remove Stored Token ZKDTSMDelegationToken_
• Could not start
• Could not update Stored Token ZKDTSMDelegationToken_
• Failed to create SecretManager

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/07481364c8f7e84e. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java:481

    } catch (Exception e) {
      LOG.error("Could not stop KeyCache", e);
    }
    try {
      if (!isExternalClient && (zkClient != null)) {
        zkClient.close();
      }
    } catch (Exception e) {
      LOG.error("Could not stop Curator Framework", e);
    }
  }

  private void createPersistentNode(String nodePath) throws Exception {
    try {
      zkClient.create().withMode(CreateMode.PERSISTENT).forPath(nodePath);
    } catch (KeeperException.NodeExistsException ne) {
      LOG.debug(nodePath + " znode already exists !!");
    } catch (Exception e) {
      throw new IOException(nodePath + " znode could not be created !!", e);
    }
  }

  @Override
  protected int getDelegationTokenSeqNum() {
    return delTokSeqCounter.getCount();
  }

  private int incrSharedCount(SharedCount sharedCount, int batchSize)
      throws Exception {
    while (true) {
      // Loop until we successfully increment the counter
      VersionedValue<Integer> versionedValue = sharedCount.getVersionedValue();
      if (sharedCount.trySetCount(
          versionedValue, versionedValue.getValue() + batchSize)) {
        return versionedValue.getValue();
      }
    }

View on GitHub (pinned to 2add963021)