{"record":{"id":"16be01e6af2c7f83","repo":"t8y2/dbx","slug":"connect-and-authenticate-to-zookeeper-s","errorCode":null,"errorMessage":"connect and authenticate to ZooKeeper: %s","messagePattern":"connect and authenticate to ZooKeeper: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"agents/drivers/argo-go/zookeeper_protocol.go","lineNumber":130,"sourceCode":"\t\tclient, err := newProtocolZooKeeperClient(connection, timeout)\n\t\tif err == nil {\n\t\t\tvar saslClient zooKeeperSASLClient\n\t\t\tsaslClient, err = newZooKeeperSASLClient(host, config)\n\t\t\tif err == nil {\n\t\t\t\terr = client.authenticateSASL(saslClient)\n\t\t\t}\n\t\t}\n\t\tif err != nil {\n\t\t\tconnection.Close()\n\t\t\tfailures = append(failures, fmt.Sprintf(\"%s: %v\", address, err))\n\t\t\tcontinue\n\t\t}\n\t\tevents := make(chan zk.Event, 1)\n\t\tevents <- zk.Event{State: zk.StateHasSession, Server: address}\n\t\tclose(events)\n\t\treturn client, events, nil\n\t}\n\treturn nil, nil, fmt.Errorf(\"connect and authenticate to ZooKeeper: %s\", strings.Join(failures, \"; \"))\n}\n\ntype protocolZooKeeperClient struct {\n\tconnection net.Conn\n\ttimeout    time.Duration\n\txid        int32\n\tmutex      sync.Mutex\n\tclosed     bool\n}\n\nfunc newProtocolZooKeeperClient(connection net.Conn, timeout time.Duration) (*protocolZooKeeperClient, error) {\n\tif connection == nil {\n\t\treturn nil, errors.New(\"ZooKeeper connection is nil\")\n\t}\n\tif timeout <= 0 {\n\t\ttimeout = defaultConnectTimeout\n\t}\n\tclient := &protocolZooKeeperClient{connection: connection, timeout: timeout}","sourceCodeStart":112,"sourceCodeEnd":148,"githubUrl":"https://github.com/t8y2/dbx/blob/c0390bff16418b651f4728520d99adf8ce48829a/agents/drivers/argo-go/zookeeper_protocol.go#L112-L148","documentation":"connectKerberosZooKeeper tries each ensemble address in turn, recording per-host failures, and returns this error when every candidate connection/authentication attempt failed. The aggregated per-host failure reasons are joined with ';' in the message, so it is the terminal 'could not establish a Kerberos-authenticated ZooKeeper session anywhere' error.","triggerScenarios":"Calling connectKerberosZooKeeper (directly or via the anonymous failover wrapper) where the inner connection factory returns failure for every address — network unreachable, TLS handshake failure, SASL/GSSAPI auth rejected, or session never established (no StateHasSession event).","commonSituations":"Kerberos principal/keytab missing or expired (kinit not run); ZooKeeper server requires SASL quorum auth the client cannot supply; firewall blocking 2181; ensemble DNS names unresolvable in the pod; test fake in TestConnectKerberosZooKeeperFailsOverAndUsesTargetHost simulating auth failure.","solutions":["Read the joined per-host failure strings in the message to find the root cause per server","Verify Kerberos credentials: run kinit with the correct principal/keytab and confirm the JAAS/GSSAPI config","Check network reachability to each host:port (telnet/nc) and DNS resolution of the ensemble names","Confirm the ZooKeeper server actually permits SASL/GSSAPI clients (authProvider configured server-side)","Retry after fixing credentials/network — the failover loop already covered all addresses"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"for _, addr := range ensemble {\n    if !isHostPort(addr) { return fmt.Errorf(\"bad ensemble address %q\", addr) }\n}\nif err := checkKerberosTicket(); err != nil { return err } // e.g. run `klist -s` equivalent before connecting","typeGuard":null,"tryCatchPattern":"client, err := connectKerberosZooKeeper(ctx, ensemble)\nif err != nil {\n    // message contains per-host failures joined by ';'\n    for _, f := range strings.Split(err.Error(), \"; \") {\n        log.Warnf(\"zk connect failure: %s\", f)\n    }\n    return fmt.Errorf(\"all ensemble members failed; verify kinit and network: %w\", err)\n}","preventionTips":["Automate kinit/keytab refresh in the service entrypoint","Health-check every ensemble member's host:port before startup","Keep the per-host failure strings from the aggregated error for diagnostics","Alert on repeated aggregate connect failures across all members"],"tags":["zookeeper","kerberos","authentication","failover"],"backgroundTag":"zookeeper-auth-failed","analyzedSha":"c0390bff16418b651f4728520d99adf8ce48829a","analyzedAt":"2026-09-05T23:05:10.900Z","contentChangedAt":"2026-09-05T23:05:10.900Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}