t8y2/dbx · error

ZooKeeper session expired during connection

Error message

ZooKeeper session expired during connection

What it means

The ZooKeeper client reported StateExpired during the connection handshake, meaning the session that was being established expired before it became usable. This is terminal for this connection attempt; the caller must reconnect with a new session.

Source

Thrown at agents/drivers/argo-go/discovery.go:212

		select {
		case <-ctx.Done():
			return ctx.Err()
		case <-timer.C:
			return errors.New("ZooKeeper connection timed out before a session was established")
		case event, ok := <-events:
			if !ok {
				return errors.New("ZooKeeper event stream closed before a session was established")
			}
			if event.Err != nil {
				return fmt.Errorf("ZooKeeper connection event: %w", event.Err)
			}
			switch event.State {
			case zk.StateHasSession:
				return nil
			case zk.StateAuthFailed:
				return errors.New("ZooKeeper authentication failed")
			case zk.StateExpired:
				return errors.New("ZooKeeper session expired during connection")
			}
		}
	}
}

func parseHiveServerRegistration(child string, data []byte) (endpoint, error) {
	candidates := []string{strings.TrimSpace(string(data)), strings.TrimSpace(child)}
	for _, candidate := range candidates {
		if candidate == "" {
			continue
		}
		if value, err := endpointFromRegistrationJSON(candidate); err == nil {
			return value, nil
		}
		parameters := parseHiveParameters(candidate)
		for _, key := range []string{"serveruri", "hiveserver2uri", "server_uri"} {
			if raw := parameter(parameters, key); raw != "" {
				return parseRegisteredEndpoint(raw)

View on GitHub (pinned to c0390bff16)

Solutions

1. Retry discovery to establish a fresh session
2. Increase the ZooKeeper session timeout in the client config
3. Fix network stability/quorum health (check server logs, ensemble latency) before retrying

Example fix

// before
zk.Connect(hosts, zk.WithSessionTimeout(2*time.Second))
// after
zk.Connect(hosts, zk.WithSessionTimeout(30*time.Second))

// prefer generous session timeouts
if cfg.ZKSessionTimeout < 10*time.Second {
	cfg.ZKSessionTimeout = 30 * time.Second
}

endpoints, err := discovery.Endpoints(ctx)
if err != nil {
	if strings.Contains(err.Error(), "session expired") {
		// transient: rebuild connection and retry with backoff
		backoff(retry)
	}
}

When it happens

Trigger: waitForZooKeeperSession receives an event with event.State == zk.StateExpired while Endpoints() is establishing the connection, e.g. long GC pauses, network partitions, or an overloaded quorum delaying the handshake past the negotiated session timeout.

Common situations: Very short negotiated session timeouts; ZooKeeper quorum under load; network instability between driver and ensemble; process pauses (GC/VM suspend) exceeding the session timeout.

Related errors

• ZooKeeper connection timed out before a session was establis
• ZooKeeper connection timed out before a session was establis
• ZooKeeper session expired during connection
• Connection timed out
• ZooKeeper SASL response is truncated

AI-assisted analysis of t8y2/dbx@c0390bff16 (2026-09-05). Data as JSON: /api/errors/fe86a0ec812d4fe9. Report an issue: GitHub.