no available HiveServer2 nodes in ZooKeeper namespace %s

Error message

no available HiveServer2 nodes in ZooKeeper namespace %s

What it means

Endpoints() found the ZooKeeper namespace (Children listed successfully) but every child resolved to zero unique endpoints without recording any failure — typically the namespace simply contains no children, or all children were filtered out as already-rejected duplicates. Distinct from the 'not found' case: the path exists but is empty (or everything was rejected).

Source

Thrown at agents/drivers/argo-go/discovery.go:160

			if parseErr == nil {
				resolved = append(resolved, value)
			} else {
				nodeFailures = append(nodeFailures, fmt.Sprintf("%s/%s: %v", path, child, parseErr))
			}
		}
		if len(resolved) > 0 {
			break
		}
	}
	resolved = shuffledEndpoints(uniqueEndpoints(resolved), rejected)
	if len(resolved) == 0 {
		if listedPath == "" {
			return nil, fmt.Errorf("HiveServer2 ZooKeeper namespace not found; tried %s", strings.Join(discovery.paths(), ", "))
		}
		if len(nodeFailures) > 0 {
			return nil, fmt.Errorf("no usable HiveServer2 nodes in ZooKeeper namespace %s: %s", listedPath, strings.Join(nodeFailures, "; "))
		}
		return nil, fmt.Errorf("no available HiveServer2 nodes in ZooKeeper namespace %s", listedPath)
	}
	return resolved, nil
}

func (discovery *zooKeeperDiscovery) paths() []string {
	namespace := strings.Trim(discovery.namespace, "/")
	if strings.EqualFold(discovery.discoveryMode, "zookeeperha") {
		return []string{
			zooKeeperPath(namespace, "instances"),
			zooKeeperPath(namespace+"-unsecure", "instances"),
			zooKeeperPath(namespace+"-sasl", "instances"),
		}
	}
	return []string{zooKeeperPath(namespace)}
}

func zooKeeperPath(parts ...string) string {
	cleaned := make([]string, 0, len(parts))

View on GitHub (pinned to c0390bff16)

Solutions

Confirm HiveServer2 instances are running and registered: zkCli ls /<namespace> should show children.
If this is a retry after connection failures, check the HiveServer2 hosts directly — they were all previously rejected as unreachable.
Wait/retry briefly if HiveServer2 is starting up; registration happens after server readiness.
Verify the same ZK ensemble is used by both HiveServer2 and the client config.

Example fix

// before: hammering a rejected set forever
endpoints, err := discovery.Endpoints(ctx, rejected) // all endpoints rejected
// after: clear rejected set after a cool-down
if len(rejected) > 0 && time.Since(lastReset) > 30*time.Second {
    rejected = map[string]bool{}
}

Defensive patterns

Strategy: fallback

Validate before calling

// preflight: namespace should contain children
conn, _, _ := zk.Connect(zkHosts, 5*time.Second)
children, _, err := conn.Children("/"+ns)
conn.Close()
if err != nil || len(children) == 0 {
    return fmt.Errorf("namespace /%s has no registered HiveServer2 instances", ns)
}

Try / catch

eps, err := discovery.Endpoints(ctx, rejected)
if err != nil && strings.Contains(err.Error(), "no available HiveServer2 nodes") {
    // fall back to a static endpoint list while ZK has nothing registered
    return staticEndpoints, nil
}

Prevention

Keep at least one HiveServer2 instance running and registered.
Clear or periodically reset the rejected-endpoint set in retry loops so healthy servers are not permanently excluded.
Add a small startup delay/retry for HiveServer2 registration lag.
Alert when the discovery znode child count drops to zero.

When it happens

Trigger: Children(path) returned an empty list, or all resolved endpoints were filtered by uniqueEndpoints/shuffledEndpoints(rejected) because every known server address was previously marked rejected.

Common situations: All HiveServer2 instances are down or just deregistered from ZK; discovery retry loop has marked every endpoint as rejected after failed connection attempts; namespace was created but HiveServer2 has not registered yet (startup race).

Related errors

AI-assisted analysis of t8y2/dbx@c0390bff16 (2026-09-05). Data as JSON: /api/errors/d6c21e3b9649f5fb. Report an issue: GitHub.