continue ZooKeeper GSSAPI negotiation at round %d: %w

Error message

continue ZooKeeper GSSAPI negotiation at round %d: %w

What it means

This error wraps a failure of the local GSSAPI implementation when computing the next SASL token from the server's challenge (saslClient.Step). authenticateSASL got a valid challenge back but the Kerberos/GSSAPI mechanism itself rejected it — e.g. the token is not valid for the current security context, the realm/principal does not match, or replay/credential problems. The client-side mechanism, not the network, failed.

Source

Thrown at agents/drivers/argo-go/zookeeper_protocol.go:226

			encoder.bytes(token)
		})
		if requestErr != nil {
			return fmt.Errorf("ZooKeeper SASL round %d: %w", round+1, requestErr)
		}
		decoder := newZooKeeperDecoder(response)
		challenge, decodeErr := decoder.bytes()
		if decodeErr != nil {
			return fmt.Errorf("decode ZooKeeper SASL round %d: %w", round+1, decodeErr)
		}
		if saslClient.Complete() {
			if len(challenge) != 0 {
				return errors.New("ZooKeeper sent an unexpected token after GSSAPI completion")
			}
			return nil
		}
		token, err = saslClient.Step(challenge)
		if err != nil {
			return fmt.Errorf("continue ZooKeeper GSSAPI negotiation at round %d: %w", round+1, err)
		}
	}
	return fmt.Errorf("ZooKeeper GSSAPI negotiation exceeded %d rounds", zooKeeperMaxSASLRounds)
}

func (client *protocolZooKeeperClient) AddAuth(scheme string, auth []byte) error {
	_, err := client.request(zooKeeperOpSetAuth, func(encoder *zooKeeperEncoder) {
		encoder.int32(0)
		encoder.string(scheme)
		encoder.bytes(auth)
	})
	return err
}

func (client *protocolZooKeeperClient) Children(path string) ([]string, *zk.Stat, error) {
	response, err := client.request(zooKeeperOpGetChildren2, func(encoder *zooKeeperEncoder) {
		encoder.string(path)
		encoder.boolean(false)

View on GitHub (pinned to c0390bff16)

Solutions

Check clock synchronization (ntp/chrony) between the client, ZooKeeper servers, and the KDC — skew over the Kerberos tolerance causes GSSAPI token rejection.
Verify the service principal the client requests matches the server's registered principal (service name and realm in krb5.conf / the connection string host reverse-resolves to the right realm).
Re-run kinit to refresh expired credentials, and ensure the credential cache path is correct for the process (CCACHE env var, keytab-based refresh for daemons).
Inspect the wrapped inner error from Step — GSSAPI major/minor status codes identify whether it is a replay, principal mismatch, or credential expiry.
If cross-realm, verify the trust path (capaths) or use a principal in the server's own realm.

Example fix

// before: principal mismatch
principal := "zkclient@WRONG.REALM" // continue ZooKeeper GSSAPI negotiation at round 1: ... principal unknown
// after: use the principal registered in the server's JAAS/keytab
principal := "zookeeper/zk-1.internal@CORRECT.REALM"

Defensive patterns

Strategy: validation

Validate before calling

// Check Kerberos prerequisites before attempting SASL
if _, err := os.Stat(os.Getenv("KRB5CCNAME")); err != nil {
    if err := exec.Command("kinit", "-kt", keytabPath, principal).Run(); err != nil {
        return fmt.Errorf("cannot acquire Kerberos credentials for %s: %w", principal, err)
    }
}
// verify clock skew
if skew := clockSkewVsKDC(); skew > 4*time.Minute {
    return fmt.Errorf("clock skew %v vs KDC exceeds Kerberos tolerance", skew)
}

Try / catch

err := client.Connect()
if err != nil && strings.Contains(err.Error(), "GSSAPI negotiation") {
    // local mechanism rejected the challenge — retrying without re-kinit won't help
    if refreshErr := kinitFromKeytab(); refreshErr != nil {
        return refreshErr
    }
    return client.Connect()
}

Prevention

Keep client and server clocks synced with NTP/chrony (Kerberos fails on >5min skew).
Verify the service principal in krb5.conf matches the server's registered ZooKeeper principal.
Use keytab-based credential refresh for long-running daemons instead of interactive tickets.
Test GSSAPI independently (kvno <service-principal>) before wiring up the client.

When it happens

Trigger: Calling authenticateSASL where saslClient.Step(challenge) returns an error at round N: server returns a GSSAPI error token (e.g. Kerberos AP_ERR replay, wrong principal in service ticket, clock skew beyond ticket skew limits), or the client's credentials expired mid-handshake.

Common situations: Clock skew between client and KDC/server exceeding the 5-minute Kerberos tolerance, the service ticket is for the wrong ZooKeeper principal (spnego/zookeeper service name mismatch in krb5.conf), credentials cache expiring during long-running agents, or cross-realm trust misconfiguration.

Related errors

AI-assisted analysis of t8y2/dbx@c0390bff16 (2026-09-05). Data as JSON: /api/errors/fa10910e43d000f8. Report an issue: GitHub.