no usable HiveServer2 nodes in ZooKeeper namespace %s: %s

Error message

no usable HiveServer2 nodes in ZooKeeper namespace %s: %s

What it means

Endpoints() successfully listed a ZooKeeper namespace (listedPath is set) but every child entry failed to yield a usable endpoint: each Get or registration parse failure was recorded in nodeFailures. The message includes the path and a semicolon-joined list of per-node failure reasons (data fetch errors or parse errors like unsupported registration format).

Source

Thrown at agents/drivers/argo-go/discovery.go:158

			}
			value, parseErr := parseHiveServerRegistration(child, data)
			if parseErr == nil {
				resolved = append(resolved, value)
			} else {
				nodeFailures = append(nodeFailures, fmt.Sprintf("%s/%s: %v", path, child, parseErr))
			}
		}
		if len(resolved) > 0 {
			break
		}
	}
	resolved = shuffledEndpoints(uniqueEndpoints(resolved), rejected)
	if len(resolved) == 0 {
		if listedPath == "" {
			return nil, fmt.Errorf("HiveServer2 ZooKeeper namespace not found; tried %s", strings.Join(discovery.paths(), ", "))
		}
		if len(nodeFailures) > 0 {
			return nil, fmt.Errorf("no usable HiveServer2 nodes in ZooKeeper namespace %s: %s", listedPath, strings.Join(nodeFailures, "; "))
		}
		return nil, fmt.Errorf("no available HiveServer2 nodes in ZooKeeper namespace %s", listedPath)
	}
	return resolved, nil
}

func (discovery *zooKeeperDiscovery) paths() []string {
	namespace := strings.Trim(discovery.namespace, "/")
	if strings.EqualFold(discovery.discoveryMode, "zookeeperha") {
		return []string{
			zooKeeperPath(namespace, "instances"),
			zooKeeperPath(namespace+"-unsecure", "instances"),
			zooKeeperPath(namespace+"-sasl", "instances"),
		}
	}
	return []string{zooKeeperPath(namespace)}
}

View on GitHub (pinned to c0390bff16)

Solutions

Inspect the nodeFailures detail in the message: fix per-node causes (usually unparseable registration payloads).
Compare the znode payload (zkCli get /ns/child) against supported formats: JSON with serverUri/server_uri/hiveServer2Uri/uri, service-record JSON, key=value params with serveruri/hiveserver2uri/server_uri, or plain host:port.
Ensure the connecting identity has read ACL on the child znodes, not just the parent.
Restart/re-register stale HiveServer2 instances that publish malformed or empty data.

Example fix

// child data is key=value with an unsupported key
// znode: port=10002
// fix: publish a recognized key
hive.server2.zookeeper.publish.configs=true  // publishes serverUri in configs

Defensive patterns

Strategy: try-catch

Validate before calling

// preflight: read one child and check it parses as host:port
conn, _, _ := zk.Connect(zkHosts, 5*time.Second)
children, _, err := conn.Children("/"+ns)
if err == nil && len(children) > 0 {
    data, _, _ := conn.Get("/" + ns + "/" + children[0])
    fmt.Printf("sample registration: %s=%q\n", children[0], data)
}
conn.Close()

Try / catch

eps, err := discovery.Endpoints(ctx, nil)
if err != nil {
    var nodeErr error
    if strings.Contains(err.Error(), "no usable HiveServer2 nodes") {
        nodeErr = err // per-node failures listed; log details for each znode
    }
    return fmt.Errorf("ZK discovery unusable: %w", err)
}

Prevention

Keep only HiveServer2 registrations in the discovery namespace.
Verify znode payload format against the driver's supported schemas (JSON serverUri, service record, key=value serveruri, host:port).
Ensure child znodes are readable by the discovery identity, not just the parent.
Pin/normalize HiveServer2's published config format across the cluster.

When it happens

Trigger: Children(path) succeeded, but for every child either connection.Get(path+"/"+child) failed (non-ErrNoNode) or parseHiveServerRegistration(child, data) returned an error, leaving resolved empty.

Common situations: HiveServer2 publishes data in a format the parser does not recognize (custom JSON schema, no serverUri/host keys, no host:port child names); transient ZK errors fetching child data; Hive instances deregistering/crashing mid-discovery; ACLs allowing Children but not Get on child znodes.

Related errors

AI-assisted analysis of t8y2/dbx@c0390bff16 (2026-09-05). Data as JSON: /api/errors/d1a7e64074845f87. Report an issue: GitHub.