ZooKeeper sent an unexpected token after GSSAPI completion

Error message

ZooKeeper sent an unexpected token after GSSAPI completion

What it means

During the ZooKeeper SASL/GSSAPI handshake the client tracks saslClient.Complete(). Once the client considers negotiation finished, the server should send an empty (zero-length) challenge as acknowledgment. A non-empty token after completion means the server is speaking an unexpected/extra SASL round or a different mechanism, so the driver aborts rather than ignoring the data.

Source

Thrown at agents/drivers/hive-go/zookeeper_protocol.go:220

	for round := 0; round < zooKeeperMaxSASLRounds; round++ {
		response, requestErr := client.request(zooKeeperOpSASL, func(encoder *zooKeeperEncoder) {
			if token == nil {
				encoder.bytes([]byte{})
				return
			}
			encoder.bytes(token)
		})
		if requestErr != nil {
			return fmt.Errorf("ZooKeeper SASL round %d: %w", round+1, requestErr)
		}
		decoder := newZooKeeperDecoder(response)
		challenge, decodeErr := decoder.bytes()
		if decodeErr != nil {
			return fmt.Errorf("decode ZooKeeper SASL round %d: %w", round+1, decodeErr)
		}
		if saslClient.Complete() {
			if len(challenge) != 0 {
				return errors.New("ZooKeeper sent an unexpected token after GSSAPI completion")
			}
			return nil
		}
		token, err = saslClient.Step(challenge)
		if err != nil {
			return fmt.Errorf("continue ZooKeeper GSSAPI negotiation at round %d: %w", round+1, err)
		}
	}
	return fmt.Errorf("ZooKeeper GSSAPI negotiation exceeded %d rounds", zooKeeperMaxSASLRounds)
}

func (client *protocolZooKeeperClient) AddAuth(scheme string, auth []byte) error {
	_, err := client.request(zooKeeperOpSetAuth, func(encoder *zooKeeperEncoder) {
		encoder.int32(0)
		encoder.string(scheme)
		encoder.bytes(auth)
	})
	return err

View on GitHub (pinned to c0390bff16)

Solutions

Verify client and server agree on the same SASL mechanism (GSSAPI/Kerberos) and versions are compatible
Check the server's SASL callback implementation for a stray final response token
Test against a single known-good ensemble member to rule out version-skew behind the proxy/ELB
Capture a wire trace of the SASL rounds to see what extra token is being sent

Example fix

// before
if saslClient.Complete() {
    if len(challenge) != 0 {
        return errors.New("ZooKeeper sent an unexpected token after GSSAPI completion")
    }
    return nil
}
// after (server-side fix: last round must return empty token)
if isFinalRound {
    return []byte{}, nil // emit empty challenge to acknowledge completion
}

Defensive patterns

Strategy: retry

Validate before calling

// Go: confirm server SASL provider is GSSAPI-compatible before connecting
// e.g. assert ensemble supports Kerberos SASL (zoo.cfg: quorum.auth.provider or jaas GSSAPI)

Try / catch

err := client.authenticateSASL(sasl)
if err != nil && strings.Contains(err.Error(), "unexpected token after GSSAPI completion") {
    if attempt < 2 {
        return reconnectAndAuthenticate(ctx) // once; else surface config/bug
    }
    return fmt.Errorf("server sent extra SASL round; check mechanism mismatch: %w", err)
}

Prevention

Keep client and ensemble ZooKeeper versions aligned
Use a standard GSSAPI JaasServer/Client config, not custom SASL callbacks
Avoid proxies between client and ensemble that transform frames

When it happens

Trigger: Server sends an additional SASL response frame after the client's GSSAPI negotiation reports Complete(); mechanism mismatch (server continuing DIGEST-MD5-style rounds while client uses GSSAPI); a misbehaving proxy inserting extra frames.

Common situations: ZooKeeper server with a SASL provider/version that emits a final token the client does not expect; custom SaslServer callback returning data on last round; mixed ensemble versions behind a load balancer.

Understand the failure class

Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

AI-assisted analysis of t8y2/dbx@c0390bff16 (2026-09-05). Data as JSON: /api/errors/cb874c98bab1d3d3. Report an issue: GitHub.