t8y2/dbx · error

ZooKeeper connection timed out before a session was establis

Error message

ZooKeeper connection timed out before a session was established

What it means

waitForZooKeeperSession waits for a connection event indicating a session was established; if the timeout timer fires first, it returns this error. The ZooKeeper client connected (or attempted to) but never reached StateHasSession within the allotted time, so discovery cannot proceed.

Source

Thrown at agents/drivers/argo-go/discovery.go:198

		if value := strings.Trim(part, "/"); value != "" {
			cleaned = append(cleaned, value)
		}
	}
	if len(cleaned) == 0 {
		return "/"
	}
	return "/" + strings.Join(cleaned, "/")
}

func waitForZooKeeperSession(ctx context.Context, events <-chan zk.Event, timeout time.Duration) error {
	timer := time.NewTimer(timeout)
	defer timer.Stop()
	for {
		select {
		case <-ctx.Done():
			return ctx.Err()
		case <-timer.C:
			return errors.New("ZooKeeper connection timed out before a session was established")
		case event, ok := <-events:
			if !ok {
				return errors.New("ZooKeeper event stream closed before a session was established")
			}
			if event.Err != nil {
				return fmt.Errorf("ZooKeeper connection event: %w", event.Err)
			}
			switch event.State {
			case zk.StateHasSession:
				return nil
			case zk.StateAuthFailed:
				return errors.New("ZooKeeper authentication failed")
			case zk.StateExpired:
				return errors.New("ZooKeeper session expired during connection")
			}
		}
	}
}

View on GitHub (pinned to c0390bff16)

Solutions

1. Increase the ZooKeeper connection timeout in the discovery config
2. Verify the ZooKeeper quorum hosts/ports are reachable (telnet/nc to 2181) and DNS resolves
3. Retry the discovery; if the quorum is down, restore it before reconnecting

Example fix

// before
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
// after
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)

timeout, err := net.DialTimeout("tcp", zkHost+":2181", 3*time.Second)
if err != nil { return fmt.Errorf("zookeeper unreachable: %w", err) }
_ = timeout.Close()

endpoints, err := discovery.Endpoints(ctx)
if err != nil {
	if errors.Is(err, context.DeadlineExceeded) || strings.Contains(err.Error(), "timed out") {
		// retry with longer timeout / backoff
	}
}

When it happens

Trigger: Endpoints() calls waitForZooKeeperSession with a context/timeout, and the events channel delivers no session event before the deadline; also hit directly in TestWaitForZooKeeperSession-style timeout tests.

Common situations: ZooKeeper quorum unreachable due to firewall/network issues; wrong host/port in the quorum string; heavily loaded ZooKeeper or DNS slowness exceeding the configured timeout.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• ZooKeeper connection timed out before a session was establis
• ZooKeeper session expired during connection
• Connection timed out
• read ZooKeeper connect response: %w
• ZooKeeper connection event: %w

AI-assisted analysis of t8y2/dbx@c0390bff16 (2026-09-05). Data as JSON: /api/errors/b54261cc35e2a01a. Report an issue: GitHub.