t8y2/dbx · error

ZooKeeper connection timed out before a session was establis

Error message

ZooKeeper connection timed out before a session was established

What it means

waitForZooKeeperSession waits for a connected/synced session event within a timeout; if the timer fires before a session is established, it aborts discovery with this error. The ZooKeeper ensemble was never reachable or never confirmed the session in time.

Source

Thrown at agents/drivers/hive-go/discovery.go:198

		if value := strings.Trim(part, "/"); value != "" {
			cleaned = append(cleaned, value)
		}
	}
	if len(cleaned) == 0 {
		return "/"
	}
	return "/" + strings.Join(cleaned, "/")
}

func waitForZooKeeperSession(ctx context.Context, events <-chan zk.Event, timeout time.Duration) error {
	timer := time.NewTimer(timeout)
	defer timer.Stop()
	for {
		select {
		case <-ctx.Done():
			return ctx.Err()
		case <-timer.C:
			return errors.New("ZooKeeper connection timed out before a session was established")
		case event, ok := <-events:
			if !ok {
				return errors.New("ZooKeeper event stream closed before a session was established")
			}
			if event.Err != nil {
				return fmt.Errorf("ZooKeeper connection event: %w", event.Err)
			}
			switch event.State {
			case zk.StateHasSession:
				return nil
			case zk.StateAuthFailed:
				return errors.New("ZooKeeper authentication failed")
			case zk.StateExpired:
				return errors.New("ZooKeeper session expired during connection")
			}
		}
	}
}

View on GitHub (pinned to c0390bff16)

Solutions

1. Verify the ZooKeeper hosts/ports are reachable (nc/zkCli smoke test)
2. Increase the discovery timeout in the connector configuration
3. Check ZooKeeper ensemble health and re-run if a leader election was in progress
4. Confirm network/firewall rules allow egress to port 2181

Example fix

// before
discovery, _ := NewZooKeeperDiscovery("zk:2181", path, time.Second) // too short
// after
discovery, _ := NewZooKeeperDiscovery("zk:2181", path, 30*time.Second)

// preflight: ensure the quorum accepts TCP connections before Endpoints
for _, host := range strings.Split(quorum, ",") {
    conn, err := net.DialTimeout("tcp", host, 3*time.Second)
    if err != nil { return fmt.Errorf("zookeeper unreachable: %s", host) }
    conn.Close()
}

endpoints, err := discovery.Endpoints(ctx)
if err != nil && strings.Contains(err.Error(), "timed out before a session") {
    // increase timeout and retry with backoff
    return retryWithBackoff(ctx, func() error { return connectWithLongerTimeout() })
}
return err

When it happens

Trigger: Calling Endpoints against an unreachable or overloaded ZooKeeper quorum where no StateHasSession event arrives before the configured timeout.

Common situations: Wrong zkQuorum host/port, network partition or firewall blocking port 2181, ZooKeeper ensemble down or in the middle of a leader election, timeout set too low for a large ensemble.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Connection timed out
• ZooKeeper connection event: %w
• ZooKeeper connection timed out before a session was establis
• ZooKeeper session expired during connection
• read ZooKeeper connect response: %w

AI-assisted analysis of t8y2/dbx@c0390bff16 (2026-09-05). Data as JSON: /api/errors/ba2f2ce0c60bf19c. Report an issue: GitHub.