ZooKeeper connection timed out before a session was…

Error message

ZooKeeper connection timed out before a session was established

What it means

waitForZooKeeperSession waits for a connection event indicating a session was established; if the timeout timer fires first, it returns this error. The ZooKeeper client connected (or attempted to) but never reached StateHasSession within the allotted time, so discovery cannot proceed.

Solutions

Increase the ZooKeeper connection timeout in the discovery config
Verify the ZooKeeper quorum hosts/ports are reachable (telnet/nc to 2181) and DNS resolves
Retry the discovery; if the quorum is down, restore it before reconnecting

Example fix

// before
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
// after
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)

Defensive patterns

Strategy: retry

Validate before calling

timeout, err := net.DialTimeout("tcp", zkHost+":2181", 3*time.Second)
if err != nil { return fmt.Errorf("zookeeper unreachable: %w", err) }
_ = timeout.Close()

Try / catch

endpoints, err := discovery.Endpoints(ctx)
if err != nil {
	if errors.Is(err, context.DeadlineExceeded) || strings.Contains(err.Error(), "timed out") {
		// retry with longer timeout / backoff
	}
}

Prevention

Pre-check ZooKeeper reachability (TCP to 2181) before discovery
Size the connection timeout above worst-case network/DNS latency
Monitor quorum health so outages are detected before clients connect

When it happens

Trigger: Endpoints() calls waitForZooKeeperSession with a context/timeout, and the events channel delivers no session event before the deadline; also hit directly in TestWaitForZooKeeperSession-style timeout tests.

Common situations: ZooKeeper quorum unreachable due to firewall/network issues; wrong host/port in the quorum string; heavily loaded ZooKeeper or DNS slowness exceeding the configured timeout.

Understand the failure class

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of t8y2/dbx@c0390bff16 (2026-09-05). Data as JSON: /api/errors/b54261cc35e2a01a. Report an issue: GitHub.

Appendix: source

Thrown at agents/drivers/argo-go/discovery.go:198

		if value := strings.Trim(part, "/"); value != "" {
			cleaned = append(cleaned, value)
		}
	}
	if len(cleaned) == 0 {
		return "/"
	}
	return "/" + strings.Join(cleaned, "/")
}

func waitForZooKeeperSession(ctx context.Context, events <-chan zk.Event, timeout time.Duration) error {
	timer := time.NewTimer(timeout)
	defer timer.Stop()
	for {
		select {
		case <-ctx.Done():
			return ctx.Err()
		case <-timer.C:
			return errors.New("ZooKeeper connection timed out before a session was established")
		case event, ok := <-events:
			if !ok {
				return errors.New("ZooKeeper event stream closed before a session was established")
			}
			if event.Err != nil {
				return fmt.Errorf("ZooKeeper connection event: %w", event.Err)
			}
			switch event.State {
			case zk.StateHasSession:
				return nil
			case zk.StateAuthFailed:
				return errors.New("ZooKeeper authentication failed")
			case zk.StateExpired:
				return errors.New("ZooKeeper session expired during connection")
			}
		}
	}
}

View on GitHub (pinned to c0390bff16)