t8y2/dbx · error

connect and authenticate to ZooKeeper: %s

Error message

connect and authenticate to ZooKeeper: %s

What it means

connectKerberosZooKeeper tries each ensemble address in turn, recording per-host failures, and returns this error when every candidate connection/authentication attempt failed. The aggregated per-host failure reasons are joined with ';' in the message, so it is the terminal 'could not establish a Kerberos-authenticated ZooKeeper session anywhere' error.

Source

Thrown at agents/drivers/argo-go/zookeeper_protocol.go:130

		client, err := newProtocolZooKeeperClient(connection, timeout)
		if err == nil {
			var saslClient zooKeeperSASLClient
			saslClient, err = newZooKeeperSASLClient(host, config)
			if err == nil {
				err = client.authenticateSASL(saslClient)
			}
		}
		if err != nil {
			connection.Close()
			failures = append(failures, fmt.Sprintf("%s: %v", address, err))
			continue
		}
		events := make(chan zk.Event, 1)
		events <- zk.Event{State: zk.StateHasSession, Server: address}
		close(events)
		return client, events, nil
	}
	return nil, nil, fmt.Errorf("connect and authenticate to ZooKeeper: %s", strings.Join(failures, "; "))
}

type protocolZooKeeperClient struct {
	connection net.Conn
	timeout    time.Duration
	xid        int32
	mutex      sync.Mutex
	closed     bool
}

func newProtocolZooKeeperClient(connection net.Conn, timeout time.Duration) (*protocolZooKeeperClient, error) {
	if connection == nil {
		return nil, errors.New("ZooKeeper connection is nil")
	}
	if timeout <= 0 {
		timeout = defaultConnectTimeout
	}
	client := &protocolZooKeeperClient{connection: connection, timeout: timeout}

View on GitHub (pinned to c0390bff16)

Solutions

1. Read the joined per-host failure strings in the message to find the root cause per server
2. Verify Kerberos credentials: run kinit with the correct principal/keytab and confirm the JAAS/GSSAPI config
3. Check network reachability to each host:port (telnet/nc) and DNS resolution of the ensemble names
4. Confirm the ZooKeeper server actually permits SASL/GSSAPI clients (authProvider configured server-side)
5. Retry after fixing credentials/network — the failover loop already covered all addresses

for _, addr := range ensemble {
    if !isHostPort(addr) { return fmt.Errorf("bad ensemble address %q", addr) }
}
if err := checkKerberosTicket(); err != nil { return err } // e.g. run `klist -s` equivalent before connecting

client, err := connectKerberosZooKeeper(ctx, ensemble)
if err != nil {
    // message contains per-host failures joined by ';'
    for _, f := range strings.Split(err.Error(), "; ") {
        log.Warnf("zk connect failure: %s", f)
    }
    return fmt.Errorf("all ensemble members failed; verify kinit and network: %w", err)
}

When it happens

Trigger: Calling connectKerberosZooKeeper (directly or via the anonymous failover wrapper) where the inner connection factory returns failure for every address — network unreachable, TLS handshake failure, SASL/GSSAPI auth rejected, or session never established (no StateHasSession event).

Common situations: Kerberos principal/keytab missing or expired (kinit not run); ZooKeeper server requires SASL quorum auth the client cannot supply; firewall blocking 2181; ensemble DNS names unresolvable in the pod; test fake in TestConnectKerberosZooKeeperFailsOverAndUsesTargetHost simulating auth failure.

Understand the failure class

• Authentication and authorization failures — expired tokens, bad credentials, and missing scopes.

Related errors

• ZooKeeper session closed because SASL authentication is requ
• ZooKeeper session closed because SASL authentication is requ
• start ZooKeeper GSSAPI negotiation: %w
• ZooKeeper SASL round %d: %w
• continue ZooKeeper GSSAPI negotiation at round %d: %w

AI-assisted analysis of t8y2/dbx@c0390bff16 (2026-09-05). Data as JSON: /api/errors/16be01e6af2c7f83. Report an issue: GitHub.