{"record":{"id":"07481364c8f7e84e","repo":"apache/hadoop","slug":"nodepath-znode-could-not-be-created","errorCode":null,"errorMessage":"${nodePath} znode could not be created !!","messagePattern":"(.+?) znode could not be created !!","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"critical","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java","lineNumber":481,"sourceCode":"    } catch (Exception e) {\n      LOG.error(\"Could not stop KeyCache\", e);\n    }\n    try {\n      if (!isExternalClient && (zkClient != null)) {\n        zkClient.close();\n      }\n    } catch (Exception e) {\n      LOG.error(\"Could not stop Curator Framework\", e);\n    }\n  }\n\n  private void createPersistentNode(String nodePath) throws Exception {\n    try {\n      zkClient.create().withMode(CreateMode.PERSISTENT).forPath(nodePath);\n    } catch (KeeperException.NodeExistsException ne) {\n      LOG.debug(nodePath + \" znode already exists !!\");\n    } catch (Exception e) {\n      throw new IOException(nodePath + \" znode could not be created !!\", e);\n    }\n  }\n\n  @Override\n  protected int getDelegationTokenSeqNum() {\n    return delTokSeqCounter.getCount();\n  }\n\n  private int incrSharedCount(SharedCount sharedCount, int batchSize)\n      throws Exception {\n    while (true) {\n      // Loop until we successfully increment the counter\n      VersionedValue<Integer> versionedValue = sharedCount.getVersionedValue();\n      if (sharedCount.trySetCount(\n          versionedValue, versionedValue.getValue() + batchSize)) {\n        return versionedValue.getValue();\n      }\n    }","sourceCodeStart":463,"sourceCodeEnd":499,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java#L463-L499","documentation":"ZKDelegationTokenSecretManager persists HA delegation-token state in ZooKeeper. During startThreads() it creates the persistent parent znodes under its namespace (/zkdtsm/ZKDTSMRoot/..., e.g. ZKDTSMTokensRoot, ZKDTSMMasterKeyRoot) via createPersistentNode(). KeeperException.NodeExistsException is tolerated (idempotent create), but every other failure - connection loss, session expiry, NoAuth, bad ACL - is rethrown as IOException \"<path> znode could not be created !!\". This aborts service startup for whatever HA service (NameNode ZKFC, ResourceManager, WebHDFS) owns the token manager.","triggerScenarios":"startThreads() -> createPersistentNode(ZK_DTSM_TOKENS_ROOT / ZK_DTSM_MASTER_KEY_ROOT / ZK_DTSM_SEQNUM_ROOT) when Curator create().forPath() fails with anything except NodeExists: ZK ensemble unreachable (bad zk-dt-secret-manager.zkConnectionString), SASL/Kerberos auth failure (zkAuthType/kerberos.* mismatch, missing JAAS Client section), znode ACL owned by a different principal from a previous deployment, connection/session timeout exhausted after zkNumRetries.","commonSituations":"First secured HA deployment without matching ZK JAAS config; /zkdtsm tree left with CREATOR_ALL_ACL by an earlier run under a different service user; typo in the ZK connect string; ZK quorum loss during HA failover; SSL-enabled ZK (zk-dt-secret-manager.ssl.*) with missing keystore/truststore config.","solutions":["Verify ZK reachability and permissions from the service host: zkCli.sh -server <quorum> ls /zkdtsm/ZKDTSMRoot (create/read/write for the service principal).","Check zk-dt-secret-manager.zkConnectionString and, for secured ZK, zk-dt-secret-manager.zkAuthType=digest|kerberos plus kerberos.keytab/kerberos.principal (or the JAAS Client login entry) match the ZK server's SASL config.","If stale ACLs block creation, stop services and delete the /zkdtsm working-path tree with zkCli so it is recreated with the current principal.","For flaky links raise zk-dt-secret-manager.zkNumRetries and zk-dt-secret-manager.zkSessionTimeout, then restart the service."],"exampleFix":"<!-- before: secured HA ZK with no auth config; create() fails NoAuth/SessionExpired -->\n<property>\n  <name>zk-dt-secret-manager.zkConnectionString</name>\n  <value>zk1:2181,zk2:2181,zk3:2181</value>\n</property>\n<!-- after: add auth matching the ZK server -->\n<property>\n  <name>zk-dt-secret-manager.zkAuthType</name>\n  <value>kerberos</value>\n</property>\n<property>\n  <name>zk-dt-secret-manager.kerberos.principal</name>\n  <value>nn/_HOST@EXAMPLE.COM</value>\n</property>\n<property>\n  <name>zk-dt-secret-manager.kerberos.keytab</name>\n  <value>/etc/security/keytabs/nn.service.keytab</value>\n</property>","handlingStrategy":"try-catch","validationCode":"// Preflight before starting the token secret manager:\n// the service principal must be able to create /zkdtsm/ZKDTSMRoot children.\ntry (CuratorFramework zk = CuratorFrameworkFactory.newClient(\n        quorum, new RetryNTimes(1, 1000))) {\n  zk.start();\n  zk.create().creatingParentsIfNeeded()\n     .withACL(ZooDefs.Ids.OPEN_ACL_UNSAFE) // match your production ACL policy\n     .forPath(\"/zkdtsm/ZKDTSMRoot\");\n  zk.delete().forPath(\"/zkdtsm/ZKDTSMRoot\"); // round-trip create/delete proves perms\n}","typeGuard":null,"tryCatchPattern":"try {\n  zkSecretManager.startThreads();\n} catch (IOException e) {\n  Throwable cause = (e.getCause() instanceof KeeperException) ? e.getCause() : e;\n  if (cause instanceof KeeperException.NoAuthException) {\n    // fix ACLs/JAAS before retrying; retry will not help\n  } else if (cause instanceof KeeperException.ConnectionLoss) {\n    // ZK unreachable: retry after restoring connectivity\n  }\n}","preventionTips":["Automate ZK ACL hygiene: always create the /zkdtsm tree with the same service principal that runs the token manager.","Add a startup dependency (systemd After=, K8s readiness) on ZooKeeper for HA services.","Smoke-test createPersistentNode equivalents (zkCli create/delete) from every service host during provisioning."],"tags":["zookeeper","delegation-token","security","ha","startup"],"backgroundTag":"zookeeper-node-creation-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}