t8y2/dbx · error

ZooKeeper session expired during connection

Error message

ZooKeeper session expired during connection

What it means

The ZooKeeper session expired while the client was still establishing its connection (StateExpired event), so waitForZooKeeperSession aborts. The ensemble considered the client dead (e.g. no heartbeat within sessionTimeout) before a usable session was confirmed.

Source

Thrown at agents/drivers/hive-go/discovery.go:212

		select {
		case <-ctx.Done():
			return ctx.Err()
		case <-timer.C:
			return errors.New("ZooKeeper connection timed out before a session was established")
		case event, ok := <-events:
			if !ok {
				return errors.New("ZooKeeper event stream closed before a session was established")
			}
			if event.Err != nil {
				return fmt.Errorf("ZooKeeper connection event: %w", event.Err)
			}
			switch event.State {
			case zk.StateHasSession:
				return nil
			case zk.StateAuthFailed:
				return errors.New("ZooKeeper authentication failed")
			case zk.StateExpired:
				return errors.New("ZooKeeper session expired during connection")
			}
		}
	}
}

func parseHiveServerRegistration(child string, data []byte) (endpoint, error) {
	candidates := []string{strings.TrimSpace(string(data)), strings.TrimSpace(child)}
	for _, candidate := range candidates {
		if candidate == "" {
			continue
		}
		if value, err := endpointFromRegistrationJSON(candidate); err == nil {
			return value, nil
		}
		parameters := parseHiveParameters(candidate)
		for _, key := range []string{"serveruri", "hiveserver2uri", "server_uri"} {
			if raw := parameter(parameters, key); raw != "" {
				return parseRegisteredEndpoint(raw)

View on GitHub (pinned to c0390bff16)

Solutions

1. Increase the ZooKeeper session timeout to exceed network round-trip latency
2. Check network stability between client and ensemble (packet loss, MTU issues)
3. Verify ensemble tickTime/sessionTimeout configuration is adequate
4. Add retry/backoff around Endpoints so transient session expiry is retried

Example fix

// before
zk.Connect(ctx, hosts, zk.WithSessionTimeout(2*time.Second)) // expires on WAN
// after
zk.Connect(ctx, hosts, zk.WithSessionTimeout(30*time.Second))

// ensure configured timeout comfortably exceeds measured RTT
rtt := measureRTT(zkHost) // e.g. via TCP connect timing
if sessionTimeout < 10*rtt {
    return fmt.Errorf("session timeout %v too small for RTT %v", sessionTimeout, rtt)
}

endpoints, err := discovery.Endpoints(ctx)
if err != nil && strings.Contains(err.Error(), "session expired") {
    // reconnect with a larger session timeout
    return retryWithNewSession(ctx, 30*time.Second)
}

When it happens

Trigger: Calling Endpoints when the ZooKeeper ensemble drops the ephemeral session during connect — network stalls, long GC pauses, or a sessionTimeout shorter than the connection latency to a distant quorum.

Common situations: Cross-region or high-latency links to ZooKeeper with a small session timeout, JVM/ZK ensemble restarts, client clock or network jitter causing missed heartbeats, tickTime too small on the ensemble.

Related errors

• ZooKeeper connection timed out before a session was establis
• ZooKeeper connection timed out before a session was establis
• Connection timed out
• read ZooKeeper connect response: %w
• ZooKeeper connection event: %w

AI-assisted analysis of t8y2/dbx@c0390bff16 (2026-09-05). Data as JSON: /api/errors/35155a750c032ab9. Report an issue: GitHub.