t8y2/dbx · error

ZooKeeper authentication failed

Error message

ZooKeeper authentication failed

What it means

ZooKeeper signaled StateAuthFailed during connection: the credentials supplied via AddAuth were rejected by the ensemble, so the session can never be authorized and discovery aborts with this error. The session exists but ACL-protected nodes will be inaccessible.

Source

Thrown at agents/drivers/hive-go/discovery.go:210

	defer timer.Stop()
	for {
		select {
		case <-ctx.Done():
			return ctx.Err()
		case <-timer.C:
			return errors.New("ZooKeeper connection timed out before a session was established")
		case event, ok := <-events:
			if !ok {
				return errors.New("ZooKeeper event stream closed before a session was established")
			}
			if event.Err != nil {
				return fmt.Errorf("ZooKeeper connection event: %w", event.Err)
			}
			switch event.State {
			case zk.StateHasSession:
				return nil
			case zk.StateAuthFailed:
				return errors.New("ZooKeeper authentication failed")
			case zk.StateExpired:
				return errors.New("ZooKeeper session expired during connection")
			}
		}
	}
}

func parseHiveServerRegistration(child string, data []byte) (endpoint, error) {
	candidates := []string{strings.TrimSpace(string(data)), strings.TrimSpace(child)}
	for _, candidate := range candidates {
		if candidate == "" {
			continue
		}
		if value, err := endpointFromRegistrationJSON(candidate); err == nil {
			return value, nil
		}
		parameters := parseHiveParameters(candidate)
		for _, key := range []string{"serveruri", "hiveserver2uri", "server_uri"} {

View on GitHub (pinned to c0390bff16)

Solutions

1. Verify auth credentials match the ACLs on the znode (getAcl via zkCli)
2. Confirm the authScheme string is exactly what the ensemble expects ("digest" vs "sasl")
3. Check the auth payload format is "user:password" for digest auth
4. Re-sync credentials after a password rotation and restart the client

Example fix

// before
discovery.authScheme = "digest"
discovery.auth = "hive:oldpassword"
// after
discovery.authScheme = "digest"
discovery.auth = "hive:newpassword" // matches znode ACL

// verify credentials against the znode ACL before discovery
func authMatchesACL(acl []zk.ACL, scheme, auth string) bool {
    cred := scheme + ":" + auth
    for _, a := range acl {
        if a.Scheme == scheme { return true }
    }
    return false
}

endpoints, err := discovery.Endpoints(ctx)
if err != nil && strings.Contains(err.Error(), "authentication failed") {
    return fmt.Errorf("check ZooKeeper authScheme/auth against znode ACLs: %w", err)
}

When it happens

Trigger: Calling Endpoints with authScheme/auth configured whose digest does not match the ACLs set on the discovery znode, producing a StateAuthFailed event in waitForZooKeeperSession.

Common situations: Wrong or rotated Hive/ZooKeeper digest password, auth configured for the wrong scheme (e.g. "digest" where ensemble expects "sasl"), typos in the user:password payload, or Kerberos/SASL mismatch.

Understand the failure class

• Authentication and authorization failures — expired tokens, bad credentials, and missing scopes.

Related errors

• ZooKeeper session closed because SASL authentication is requ
• ZooKeeper session closed because SASL authentication is requ
• username is required when auth_scheme = "sasl_digest"
• password is required when auth_scheme = "sasl_digest"
• ZooKeeper authentication failed

AI-assisted analysis of t8y2/dbx@c0390bff16 (2026-09-05). Data as JSON: /api/errors/5fdef8c04e145e25. Report an issue: GitHub.